This chapter covers the development of the Java API calls and the configuration required to replicate an IBM FileNet Document Management Object Store.
See also:
www.ibm.com/docs/en/filenet-p8-platform/5.2.1?topic=guide-replication
Although this link states:
“Each object in a Content Engine object store has a unique ID that is controlled by the Content Engine. However, a replicated object in an external repository possesses a different unique ID from the original object over which the Content Engine has no control. An ExternalIdentity object represents the identity of a replicated object in an external repository.”
I will demonstrate a Java program process which will maintain an exact replication of the original Object Store ID Documents and their containing folder structure. Unlike the standard replication discussed in the preceding link, the Java Replication program covered in this chapter is designed to even copy the unique GUIDs of the original IBM FileNet Content Object Store Objects (including Documents, Folders, and Property objects).
This functionality is useful for larger organizations with geographically separate sites, where the Standard Operating Procedures and Quality Manuals have to be consistently maintained, but usually require a single point of authorship to maintain the same standards of production across all the sites.
Chapter Organization
This chapter contains the following six core Parts, each of which is organized by sections within:
Part 1 – Bill of Materials. This Part lists the prerequisite IBM Software components, including the IBM product code numbers, required to download the latest IBM FileNet Content Manager software components, used to test the Java Replication Program FileNet API calls. It also covers the download of the document describing the installation of the software on a Red Hat Linux server.
Part 2 – The Replication Program Introduction. This part describes the main Replication Program functionality and the development tools and procedure.
Part 3 – Code Developed for the Replication Program. This Part lists the main Replication Program Java Code.
Part 4 – Code Listing – CEReplicateConfig. In this Part, the Java Code for the supporting utility methods is listed.
Part 5 – Code Listing – CEMigrateConfig. In this Part, the supporting Java Code for the CEMigrateConfig code is listed. This code reads the parameters from the config.xml file which defines the source and target object stores and the required login and security parameters.
Part 6 – Code Listing – Supporting Utility Code. In this Part, the supporting utility Java Code for the Replication Program is listed. This is defined for the compiled fn_utils.jar, fn_connect.jar, and the fn_connection.jar supporting libraries.
Part 1 – Bill of Materials
The installation steps required for the base IBM Case Manager 5.3.3 system we are using are covered step by step in the following free ResearchGate document, downloaded using the DOI URL as follows:
https://doi.org/10.13140/RG.2.2.21708.16001
entitled “Case Manager 5.3.3 Installation on RHEL 8.0 with Content Navigator 3.0.6”
Click the Download file PDF command button in Figure 4-1.
A screenshot depicts a file titled case manager to install R H E L version 8.0 content navigator 3.0.6 I F 003, authored by Alan Bluck. There is an option to download the file in P D F format.
Figure 4-1
Download the IBM Case Manager Installation Document
This gives the downloaded document entitled CaseManagerInstallationonRHEL8.0_V3.docx.
The installed IBM software product codes required are listed as follows:
IBM FileNet Content Manager V5.5.0 for IBM Case Foundation V5.3.0 Multiplatform Multilingual eAssembly (CJ2VNML)
FileNet Content Manager V5.5.0 Quick Start Guide Multiplatform Multilingual (CNP8XML)
IBM FileNet Content Platform Engine V5.5.0 Linux Multilingual (CNP8ZML)
IBM FileNet Content Platform Engine V5.5.0 Windows Multilingual (CNP90ML)
IBM FileNet Content Platform Engine Client V5.5.0 Linux English (CNP93EN)
IBM FileNet Content Platform Engine Client V5.5.0 Windows English (CNP94EN)
IBM DB2 Enterprise Server Edition Restricted Use Quick Start and Activation V11.1 for Linux, UNIX and Windows Multilingual (CNB25ML)
Quick Start Guide for IBM WebSphere Application Server V9.0 (CNA8LML)
IBM WebSphere Application Server V9.0 (CND1AML)
IBM WebSphere Application Server V9.0 Supplements – Application Client (CND1CML)
IBM WebSphere Application Server V9.0 Supplements – IBM HTTP Server (CND1DML)
IBM WebSphere Application Server V9.0 Supplements – Web Server Plugins (CND1EML)
IBM WebSphere Application Server V9.0 Supplements – WebSphere Customization Toolkit (CND1FML)
IBM WebSphere Application Server Liberty (IBM Installation Manager install) (CND1GML)
IBM Installation Manager V1.8.5 for Linux x86_64 (CND0ZML)
IBM SDK, Java (TM) Technology Edition, Version 8 for Windows (CND15ML)
IBM SDK, Java (TM) Technology Edition, Version 8 for Linux (CND18ML)
Note
WebSphere 8.5.5 Fix pack 15 automatically installs and defaults the JDK to 1.8.
Part 2 – The Replication Program Introduction
This Java program consists of 4116 lines of Java code calling methods from the FileNet 5.x Java APIs to replicate documents and folders from a selected Folder location in a source Object Store to a selected Target Object Store on a different server. (See the section code lists in this chapter.)
The Target documents are copied with exact GUIDS, Version Series ID, and Security profile and folder linkage as in the source Object Store, except that the Group and User security are set to Read Only for the Replicated Folder and Document Properties and Content.
One exception Group and User are left with their originally defined security levels.
The Replication program also detects security and property changes in the Source Object store and the removal/move of Documents and Folders and faithfully copies the changes to the target object store.
This program was built to satisfy the following requirements:
a) It can be rerun from any starting date and updates the Target replica with all changes; there is no need to reset the target object store as the program has been designed to check and ignore existing Documents or Folders already replicated.
b) The tests for specifications and functionality of the program are as follows:
- 1.First replication:
Verify the object store and folders.
The same object store and folders are found at target.
- 2.Verify the document at the target side.
The number of documents are the same.
- 3.Verify the permission setup.
The permission setups are the same.
- 4.Check whether the copy at the target side is editable or not.
The documents on the target Object Store must be READ-ONLY.
- 5.Verify the document change – Upload a new document at the source.
A new document should be synced to the target, with identical permissions.
- 6.Upload a new version of the document at the source.
A new version of the document should be found at the target.
- 7.Change the permissions of the document at the source.
The permissions must be updated at the target.
(Within the READONLY specifications!)
- 8.Remove a document at the source.
The same document should be removed at the target.
- 9.Modify the document attribute.
The same attribute change is applied at the target.
- 10.Move the document from one folder to another folder at the source.
The same change is applied at the target.
- 11.Change the replication frequency.
The target copy should be refreshed as per the frequency setting.
- 12.Verify the folder change – Create a new folder at the source.
A new folder should be synced to the target, with identical permissions.
- 13.Verify the folder change – Amend the name of an existing folder at the source.
The Folder Name change should be synced to the target Object Store.
- 14.Remove a folder at the source Object Store.
The same folder should be removed at the target Object Store.
- 15.Verify the target Object Store copy is still accessible if the source Object Store copy is stopped.
The target copy should still be accessible for read.
Nonfunctional Requirements
- 1)
An administrator should be able to adjust the replication frequency.
(Originally defined as 1 hour, but run at present every 15 minutes)
- 2)
The full Java code source is in Part 6 of this chapter.
- 3)
The Replication script should be compatible for future FileNet 5.x versions. (All API calls are compatible with the base FileNet 5.2 version.)
Development Tools Used
- 1)
VMware Red Hat Enterprise Linux Server RHEL 8.0 with full IBM FileNet P8 5.5.5 system
A screenshot depicts R H E L 8.0 with information related to devices and descriptions. A tab of activities includes project explorer includes 3 folders.
Figure 4-2
The Red Hat Linux server specification used for the development
A screenshot depicts product information with version, license information with status, type, and expiration details, and additional information with memory and file names.
Figure 4-3
The VMware Host server and version details
The P8 5.5.5 Environment
Machine Name: ECMUKDEMO6
Red Hat Linux Enterprise 8.0 Server
Main User: Alan
WebSphere User: wasadm
DB2 User: db2inst1
Installed WebSphere 8.5.5.15
Installation Manager 1.8.5
Installation IBM Security Directory Services 6.4
Installation DB2 Enterprise Server 11.5
Product name: "IBM Data Server Client"
Product identifier: "db2client"
Version information: "11.5"
Product name: "IBM DB2 Developer-C Edition"
License type: "Community"
Expiry date: "Permanent"
Product identifier: "db2dec"
Version information: "11.5"
Max amount of memory (GB): "16"
Max number of cores: "4"
Max amount of table space (GB): "100"
Login: root
IBM FileNet Case Manager 5.5.3
IBM FileNet Content Engine 5.5.5
Eclipse IDE Version 2021-12 (4.22.0) for Red Hat Linux 8.0
An installation box of the eclipse I D E depicts the version and build I D software downloading sources, and tools with an option to close at the bottom right.
Figure 4-4
The Eclipse IDE Java version used for the development in this book
(Download eclipse-java-2022-03-R-linux-gtk-x86_64.tar.gz for unpacking and installation.)
At IBM FileNet Content Engine 5.x, there are Java packages and classes which help with the Java API calls used in the Replication code, especially as follows:
com.ibm.ce.rep.utils.CEReplicate
com.ibm.ce.utils.CEReplicateConfig
New Configuration Files:
(These can all be found in the /opt/IBM/FileNet/CEClient folder path.)
WcmApiConfig_ejb.properties
WcmApiConfig_wsi.properties
Unit Test Data
The Root “/AUDIT_TEST” folder in the OS2 “Target” object store was used for initial testing, Test Data, as shown in Figure 4-5.
A window depicts object store O S 2 with options to browse, root folder, audit test, level 2 A, 1 B, and 1 C files. A document named Doc level 1 A is listed under level 1 A folder with creation date and author name.
Figure 4-5
The Test Data folder structure and example Document for the first tests
Project references required for the Java Replication program
AUDIT_Common
AUDIT_Utils
FileNetConnection
FN_Connect
Folder Referenced Object properties replicated
- a)
Permissions (Security ACLs)
- b)
Containers
- c)
Containees
- d)
SubFolders
- e)
Contained Documents (retrieved separately in the importDocuments method)
Folder Referenced Object properties NOT replicated
- a)
Replication Group
- b)
External Replica Identities
- c)
Active Markings
- d)
Annotations
- e)
Security Policy
- f)
Coordinated Tasks
- g)
Workflow Subscriptions
Note
The following is required to write DateCreated and Creator properties, etc.
Creator Property
See the following links for updating this:
www.ibm.com/docs/en/filenet-p8-platform/5.5.0?topic=security-working
This indicates the name of the user assigned as the creator of the object.
www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=comfilenetapiconstants-accessright
Settability of this property is read-only for most users. For users who have been granted privileged write access (AccessRight.PRIVILEGED_WRITE), this property is settable only on create. After initial object creation, this property is read-only for all users.
Setting Object Store Access Rights
The permissions (access rights) associated with an object store control the degree of access that users have to the objects within the object store. Granting full control means that the specified user is granted permission to connect to the object store, store objects, modify objects, and remove objects. (For additional examples, see Document security levels – IBM Documentation, www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=rights-document-security-levels.)
An illustration depicts the static and privileged write scripts in two columns. The user or group is granted or denied permission to set the system are specified.
Figure 4-6
The static AccessRight privilege we need to keep the same system properties
Shell Script Batch Jobs
The following Shell Script jobs were developed to run a scheduled replication service using the Linux cron system to run the Java Replication program which we develop. For cron details, see www.redhat.com/sysadmin/automate-linux-tasks-cron.
For 15-minute repeats of the Replication, we use
*/15 * * * * /opt/replication/Replicate.sh
The following also works:
0,15,30,45 * * * * /opt/replication/Replicate.sh
Shell Script Name: Replicate.sh
#!/bin/sh
# This command file launches the Replication application.
# Java copied to the path /opt/replication/applibs/sdk/bin
STARTING_FOLDER=/opt/replication
export STARTING_FOLDER
CLASSPATH=${STARTING_FOLDER}/jars/replication.jar:${STARTING_FOLDER}/jars/Jace.jar:${STARTING_FOLDER}/jars/log4j-1.2.17.jar
export CLASSPATH
JAVA_HOME=${STARTING_FOLDER}/applibs/sdk
export JAVA_HOME
JAVA_BIN=${JAVA_HOME}/bin
export JAVA_BIN
source ${STARTING_FOLDER}/envCommonReplicate.sh
source ${STARTING_FOLDER}/env5x.sh
echo CLASSPATH= ${CLASSPATH}
echo Starting Replication job ... Output to console.log in this directory
${JAVA_BIN}/java com.ibm.ce.rep.utils.CEReplicate > console.log
A screenshot depicts 15 lines of commands under address forward slash, opt, forward slash, replication to launch the replication application.
Figure 4-6A
The main Replication.sh shell script for running in the cron scheduled job system
Shell Script Name: envCommonReplicate.sh
#!/bin/sh
# This command file sets the CLASSPATH environment for the
# Replication application
STARTING_FOLDER=/opt/replication
export STARTING_FOLDER
export APPSLIBS=${STARTING_FOLDER}/libs/applibs/
export OTHER=${STARTING_FOLDER}/libs/other/
CLASSPATH=${APPSLIBS}replication.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${APPSLIBS}fn_common.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${APPSLIBS}fn_connection.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${APPSLIBS}fn_utils.jar;
export CLASSPATH
CLASSPATH=${CLASSPATH}:${APPSLIBS}fn_connect.jar;
export CLASSPATH
CLASSPATH=${CLASSPATH}:./config
export CLASSPATH
A screenshot depicts 15 lines of command to set the classpath environment for the replication environment. The commands start with a start folder and end with envCommonReplicate dot S H.
Figure 4-6B
The CLASSPATH environment is set using this shell script
Shell Script Name: env5x.sh
#!/bin/sh
# This command file launches the IBM FileNet library CLASSPATH settings application.
STARTING_FOLDER=/opt/replication
export STARTING_FOLDER
FIVEXLIBS=${STARTING_FOLDER}/libs/5xlibs/
export FIVEXLIBS
CLASSPATH=${CLASSPATH}:${FIVEXLIBS}Jace.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${FIVEXLIBS}log4j.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${FIVEXLIBS}log4j-1.2.17.jar
export CLASSPATH
CLASSPATH=${CLASSPATH}:${FIVEXLIBS}xerces.jar
export CLASSPATH
# jars for Encryption 1.6
CLASSPATH=${CLASSPATH}:${FIVEXLIBS}commons-codec-1.15.jar
export CLASSPATH
A screenshot depicts 15 lines of command to launch the I B M File Net library classpath. The commands start with a start folder and end with env5x dot s h.
Figure 4-6C
The IBM FileNet API jar files added to the CLASSPATH with the env5x.sh shell script
The edited shell script files have the Linux security updated as shown in Figure 4-6D.
10 line of command illustrates the square bracket root at E C M U K D E M 0 6 replication, square bracket, with application library env5x dot s h and envCommonReplicate dot S H.
Figure 4-6D
The security on the shell script files is updated using chmod 755 *.sh
Linux Directory Paths Required
The following library paths are created for holding the Replication program Java .jar library files:
/opt/replication
/opt/replication/config
10 line of command illustrates the square bracket, root at E C M U K D E M 0 6 replication, square bracket, with config application library env5x dot s h and envCommonReplicate dot S H.
Figure 4-6E
The /opt/replication/config subdirectory is created and config.xml copied from Eclipse
/opt/replication/libs
/opt/replication/jars
/opt/replication/jars/sdk
/opt/replication/jars/sdk/bin
/opt/replication/libs/5xlibs
/opt/replication/libs/other
The Replication.jar file program uses the following .jars on the AUDIT_CEReplication Project Build Path.
A screenshot depicts 9 class folders J A R program files with a path under the J R E system library.
Figure 4-7
The supporting .jar files required by the main Replication.jar program
A command illustrates the square bracket, root at E C M U K D E M 0 6 replication, square bracket, with app libs logs and app libs jar logs.
Figure 4-7A
The /opt/replication jar library folders are created
(base) [root@ECMUKDEMO6~]# cd /opt/replication
(base) [root@ECMUKDEMO6 replication]# pwd
/opt/replication
(base) [root@ECMUKDEMO6 replication]# mkdir applibs
(base) [root@ECMUKDEMO6 replication]# ls
applibs logs
(base) [root@ECMUKDEMO6 replication]# mkdir jars
(base) [root@ECMUKDEMO6 replication]# ls
applibs jars logs
(base) [root@ECMUKDEMO6 replication]# mkdir libs
(base) [root@ECMUKDEMO6 replication]# cd libs
A command illustrates the square bracket, root at E C M U K D E M 0 6 replication, square bracket, with applibs logs and applibs other.
Figure 4-7B
The /opt/replication/libs jar library folders are created
The following files are copied into the folders created earlier:
(base) [root@ECMUKDEMO6 libs]# pwd
/opt/replication/libs
(base) [root@ECMUKDEMO6 libs]# ls
(base) [root@ECMUKDEMO6 libs]# mkdir applibs
(base) [root@ECMUKDEMO6 libs]# mkdir other
(base) [root@ECMUKDEMO6 libs]# ls
applibs other
(base) [root@ECMUKDEMO6 libs]#
A screenshot depicts commands under forward slash opt forward slash replication forward slash applibs to copy the supporting jar files common, connection, connect, event handlers, and utils into the directory.
Figure 4-7C
The supporting jar files copied to the /opt/application/applibs subdirectory
In summary, the folders are created as follows:
/opt/replication
/opt/replication/config
/opt/replication/libs
/opt/replication/jars
/opt/replication/jars/sdk
/opt/replication/jars/sdk/bin
/opt/replication/libs/5xlibs
/opt/replication/libs/applibs
/opt/replication/libs/other
Jar files and shell script files are located as follows:
/opt/replication/Replicate.sh
/opt/replication/envCommonReplicate.sh
/opt/replication/env5x.sh
/opt/replication/log4j.dtd
/opt/replication/log4j.xml
/opt/replication/config/config.xml
/opt/replication/jars/replication.jar
/opt/replication/jars/Jace.jar
/opt/replication/jars/log4j-1.2.17.jar
/opt/replication/jars/log4j.jar
/opt/replication/jars/eeapi.jar
/opt/replication/jars/
/opt/replication/libs/applibs/replication.jar
/opt/replication/libs/applibs/fn_common.jar
/opt/replication/libs/applibs/fn_connection.jar
/opt/replication/libs/applibs/fn_utils.jar
/opt/replication/libs/applibs/fn_connect.jar
/opt/replication/libs/5xlibs/Jace.jar
/opt/replication/libs/5xlibs/log4.jar
/opt/replication/libs/5xlibs/xerces.jar
/opt/replication/libs/5xlibs/commons-codec-1.15.jar
Jars Required for Content Engine Client
A web page depicts the path under objects under O S 2 as data design, classes, and folders. An audit definition and the option to create a new one are highlighted under the folder section.
Figure 4-7D
The list of supporting IBM FileNet P8 Content Engine Version 5.5.x jars
The files are copied as follows:
(base) [root@ECMUKDEMO6 libs]# mkdir 5xlibs
(base) [root@ECMUKDEMO6 libs]# ls
5xlibs applibs other
(base) [root@ECMUKDEMO6 libs]# cd 5xlibs/
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/Jace.jar .
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/commons-codec-1.15.jar .
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/log4j.* .
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/xerces.jar .
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/pe*.jar .
(base) [root@ECMUKDEMO6 5xlibs]# cp /opt/FileNetJars/eeapi.jar .
(base) [root@ECMUKDEMO6 5xlibs]# chmod 755 xerces.jar
(base) [root@ECMUKDEMO6 5xlibs]# chmod 755 commons-codec-1.15.jar
(base) [root@ECMUKDEMO6 5xlibs]# pwd
/opt/replication/libs/5xlibs
(base) [root@ECMUKDEMO6 5xlibs]#
For /opt/replication/libs/applibs:
(base) [root@ECMUKDEMO6 libs]# cd applibs
(base) [root@ECMUKDEMO6 applibs]# ls
(base) [root@ECMUKDEMO6 applibs]# cp /root/eclipse/java-2021-12/eclipse/fn*.jar .
(base) [root@ECMUKDEMO6 applibs]# cp /root/eclipse/java-2021-12/eclipse/replication.jar .
(base) [root@ECMUKDEMO6 applibs]# ls
fn_common.jar fn_connection.jar fn_connect.jar fn_eventhandlers.jar fn_utils.jar replication.jar
(base) [root@ECMUKDEMO6 applibs]# pwd
/opt/replication/libs/applibs
(base) [root@ECMUKDEMO6 applibs]#
For /opt/replication/jars:
(base) [root@ECMUKDEMO6 replication]# cd jars
(base) [root@ECMUKDEMO6 jars]# ls
(base) [root@ECMUKDEMO6 jars]# cp /root/eclipse/java-2021-12/eclipse/replication.jar .
(base) [root@ECMUKDEMO6 jars]# cp /opt/FileNetJars/Jace.jar .
(base) [root@ECMUKDEMO6 jars]# cp /opt/FileNetJars/log4j*.jar .
(base) [root@ECMUKDEMO6 jars]# cp /opt/FileNetJars/eeapi.jar .
(base) [root@ECMUKDEMO6 jars]# ls
eeapi.jar Jace.jar log4j-1.2.17.jar log4j.jar replication.jar
(base) [root@ECMUKDEMO6 jars]# chmod 755 replication.jar
(base) [root@ECMUKDEMO6 jars]#
(base) [root@ECMUKDEMO6 jars]# ls
eeapi.jar Jace.jar log4j-1.2.17.jar log4j.jar replication.jar
(base) [root@ECMUKDEMO6 jars]# pwd
/opt/replication/jars
(base) [root@ECMUKDEMO6 jars]#
The Java sdk used by Case Manager is copied to the /opt/replication/applibs subfolder using the command:
cp -R /opt/IBM/CaseManagement/java/ /opt/replication/applibs
Config Area
(base) [root@ECMUKDEMO6 replication]# cd config
(base) [root@ECMUKDEMO6 config]# ls -lsa
total 4
0 drwxr-xr-x. 2 root root 24 Aug 1 15:25 .
0 drwxr-xr-x. 7 root root 191 Aug 2 04:53 ..
4 -rwxr-xr-x. 1 root root 2495 Aug 1 15:16 config.xml (NB Date Modified and Size is updated every run!)
(base) [root@ECMUKDEMO6 config]# pwd
/opt/replication/config
(base) [root@ECMUKDEMO6 config]#
Encryption of the Administrator User Password in config.xml by the Program
The processing is as follows:
- a.
Just for the first run, the Config.xml password setting is unencrypted.
- b.
The first run of the Replication program will load the unencrypted password (remember this is the exact “clear” unencrypted password string and so is a vulnerable security risk just for the first run of the program!)
- c.
It will attempt to decrypt the string found.
- d.
It will test the decrypted string is wrapped as follows: Encrypt3dpasswordvalueP4ssw0rd
- e.
If the decrypted string does not contain the start and end strings shown earlier, it will assume the config contains a clear password.
- f.
In this case, the program will wrap the start and end strings Encrypt3d and P4ssw0rd around the input clear password string.
- g.
Encrypt the resulting Encrypt3dpasswordvalueP4ssw0rd string and write out the result to the config.xml file.
- h.
If the decrypted string is in the form Encrypt3dpasswordvalueP4ssw0rd, passwordvalue will be set as the password.
- i.
Then the read in password will be set in the replication program for use.
Update of the Config.xml Start Date
At the point in time the replication program is started, the current run date and time from the local server system clock will be used as the next start date/time and written back to the config .xml file in the following format:
20220719T114828Z
NB Delta set at -1 indicates no overlap.
0 is 1 hour prior to last run date (to the nearest whole hour).
1 is 2 hours prior "" etc.
Config.xml
This was renamed on Linux as config.xml (all lowercase).
Note here that the source Object Store is defined in the <ceimport> tag of the XML!
<?xml version="1.0" encoding="UTF-8"?><config>
<ceimport>
<osname>OS1</osname>
<osrootfolder>/AUDIT_TEST</osrootfolder>
<ceuser>Alan</ceuser>
<cepassword>RW5jcnlwdDNkZmlsZW5ldFA0c3N3MHJk</cepassword>
<ceurl>http://ecmukdemo6:9080/wsi/FNCEWS40MTOM/</ceurl>
<celoginconfig>/opt/IBM/FileNet/CEClient/config/jaas.conf.WSI</celoginconfig>
<cestanza>FileNetP8WSI</cestanza>
<UpdatingBatchSize>250</UpdatingBatchSize>
<MaxDocCount>50</MaxDocCount>
<MaxRunTimeMinutes>30</MaxRunTimeMinutes>
<MaxConsecutiveErrors>1000</MaxConsecutiveErrors>
<AuditFile>/opt/replication/logs/AuditImportAuditDocs.log</AuditFile>
<AuditFileFolders>/opt/replication/logs/AuditImportAuditFolders.log</AuditFileFolders>
</ceimport>
<cereplicate>
<MaxSearchSize>2000</MaxSearchSize>
<readonlygroup>remoteauditors</readonlygroup>
<startsearchdate>20220801T151626Z</startsearchdate>
<DeleteAuditFile>/opt/replication/logs/AuditImportAuditDeleteDocs.log</DeleteAuditFile>
<deleteMaxDocCount>500</deleteMaxDocCount>
<DeleteFolderAuditFile>/opt/replication/logs/AuditImportAuditDeleteFolders.log</DeleteFolderAuditFile>
<DeleteMaxFolderCount>500</DeleteMaxFolderCount>
<excludedgroup>p8admins</excludedgroup>
<excludeduser>Alan</excludeduser>
<UpdateDocumentAuditFile>/opt/replication/logs/AuditImportAuditUpdateDocuments.log</UpdateDocumentAuditFile>
<UpdateMaxDocCount>500</UpdateMaxDocCount>
<MaxConsecutiveUpdateErrors>500</MaxConsecutiveUpdateErrors>
<DebugOutputFlag>off</DebugOutputFlag>
<LDAPSearchFlag>on</LDAPSearchFlag>
<MaxConsecutiveDeleteErrors>500</MaxConsecutiveDeleteErrors>
<DeltaHours>-1</DeltaHours>
<FolderSubclasses>Folder,FOLDER_ATTRIBUTE</FolderSubclasses>
</cereplicate>
<ceexport>
<osname>OS2</osname>
<ceuser>Alan</ceuser>
<cepassword>RW5jcnlwdDNkZmlsZW5ldFA0c3N3MHJk</cepassword>
<MaxDocCount>500</MaxDocCount>
<MaxRunTimeMinutes>0</MaxRunTimeMinutes>
<MaxConsecutiveErrors>1000</MaxConsecutiveErrors>
<AuditFile>/opt/replication/logs/AuditExportAudit.log</AuditFile>
<SQLBatchSize>250</SQLBatchSize>
<celoginconfig>/opt/IBM/FileNet/CEClient/config/jaas.conf.WSI </celoginconfig>
<ceurl>http://ecmukdemo6:9080/wsi/FNCEWS40MTOM</ceurl>
<cestanza>FileNetP8WSI</cestanza>
</ceexport>
log4j.xml (Updated to Fix Error Issues)
The first run of the Replication program produced the following two errors:
log4j:ERROR No appender named [ rfa_threads ] could be found.
log4j:ERROR No appender named [rfa2] could be found.
In the following XML, the new appender sections are highlighted:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd" >
<log4j:configuration>
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="DEBUG" />
<param name="levelMax" value="INFO" />
</filter>
</appender>
<appender name="rfa" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/filenet_app.log"/>
<param name="MaxFileSize" value="100KB"/>
<!-- Keep some backup files -->
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%p %t %c - %m%n"/>
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="WARN" />
<param name="levelMax" value="FATAL" />
</filter>
</appender>
<appender name="rfa2" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/filenet_app2.log"/>
<param name="MaxFileSize" value="100KB"/>
<!-- Keep some backup files -->
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%p %t %c - %m%n"/>
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="WARN" />
<param name="levelMax" value="FATAL" />
</filter>
</appender>
<appender name=" rfa_threads " class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/filenet_appthreads.log"/>
<param name="MaxFileSize" value="100KB"/>
<!-- Keep some backup files -->
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%p %t %c - %m%n"/>
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="WARN" />
<param name="levelMax" value="FATAL" />
</filter>
</appender>
<logger name="com.ibm.filenet.ce" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name="filenet_error.api.com.filenet.apiimpl.util.ConfigValueLookup" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name="filenet_tracing.api.detail.moderate.summary.com.filenet.apiimpl.util.ConfigValueLookup" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name=" com.ibm.ce.utils.FileCopier " additivity="false">
<level value="debug" />
<appender-ref ref=" rfa_threads " />
</logger>
<logger name="com.ibm.ce.rep.utils.CEReplicate" additivity="false">
<level value="info" />
<appender-ref ref="rfa2" />
<appender-ref ref="stdout" />
</logger>
</log4j:configuration>
Original log4j.xml File
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd" >
<log4j:configuration>
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="DEBUG" />
<param name="levelMax" value="INFO" />
</filter>
</appender>
<appender name="rfa" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/filenet_app.log"/>
<param name="MaxFileSize" value="100KB"/>
<!-- Keep some backup files -->
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%p %t %c - %m%n"/>
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="WARN" />
<param name="levelMax" value="FATAL" />
</filter>
</appender>
<logger name="com.ibm.filenet.ce" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name="filenet_error.api.com.filenet.apiimpl.util.ConfigValueLookup" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name="filenet_tracing.api.detail.moderate.summary.com.filenet.apiimpl.util.ConfigValueLookup" additivity="false">
<level value="debug" />
<appender-ref ref="rfa" />
<appender-ref ref="stdout" />
</logger>
<logger name=" com.ibm.ce.utils.FileCopier " additivity="false">
<level value="debug" />
<appender-ref ref=" rfa_threads " />
</logger>
<logger name="com.ibm.ce.rep.utils.CEReplicate" additivity="false">
<level value="info" />
<appender-ref ref="rfa2" />
<appender-ref ref="stdout" />
</logger>
</log4j:configuration>
log4j.dtd
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Authors: Chris Taylor, Ceki Gulcu. -->
<!-- Version: 1.2 -->
<!-- A configuration element consists of optional renderer
elements,appender elements, categories and an optional root
element. -->
<!ELEMENT log4j:configuration (renderer*, appender*,plugin*, (category|logger)*,root?,
(categoryFactory|loggerFactory)?)>
<!-- The "threshold" attribute takes a level value below which -->
<!-- all logging statements are disabled. -->
<!-- Setting the "debug" enable the printing of internal log4j logging -->
<!-- statements. -->
<!-- By default, debug attribute is "null", meaning that we not do touch -->
<!-- internal log4j logging settings. The "null" value for the threshold -->
<!-- attribute can be misleading. The threshold field of a repository -->
<!-- cannot be set to null. The "null" value for the threshold attribute -->
<!-- simply means don't touch the threshold field, the threshold field -->
<!-- keeps its old value. -->
<!ATTLIST log4j:configuration
xmlns:log4j CDATA #FIXED "http://jakarta.apache.org/log4j/"
threshold (all|trace|debug|info|warn|error|fatal|off|null) "null"
debug (true|false|null) "null"
reset (true|false) "false"
>
<!-- renderer elements allow the user to customize the conversion of -->
<!-- message objects to String. -->
<!ELEMENT renderer EMPTY>
<!ATTLIST renderer
renderedClass CDATA #REQUIRED
renderingClass CDATA #REQUIRED
>
<!-- Appenders must have a name and a class. -->
<!-- Appenders may contain an error handler, a layout, optional parameters -->
<!-- and filters. They may also reference (or include) other appenders. -->
<!ELEMENT appender (errorHandler?, param*,
rollingPolicy?, triggeringPolicy?, connectionSource?,
layout?, filter*, appender-ref*)>
<!ATTLIST appender
name CDATA #REQUIRED
class CDATA #REQUIRED
>
<!ELEMENT layout (param*)>
<!ATTLIST layout
class CDATA #REQUIRED
>
<!ELEMENT filter (param*)>
<!ATTLIST filter
class CDATA #REQUIRED
>
<!-- ErrorHandlers can be of any class. They can admit any number of -->
<!-- parameters. -->
<!ELEMENT errorHandler (param*, root-ref?, logger-ref*, appender-ref?)>
<!ATTLIST errorHandler
class CDATA #REQUIRED
>
<!ELEMENT root-ref EMPTY>
<!ELEMENT logger-ref EMPTY>
<!ATTLIST logger-ref
ref CDATA #REQUIRED
>
<!ELEMENT param EMPTY>
<!ATTLIST param
name CDATA #REQUIRED
value CDATA #REQUIRED
>
<!-- The priority class is org.apache.log4j.Level by default -->
<!ELEMENT priority (param*)>
<!ATTLIST priority
class CDATA #IMPLIED
value CDATA #REQUIRED
>
<!-- The level class is org.apache.log4j.Level by default -->
<!ELEMENT level (param*)>
<!ATTLIST level
class CDATA #IMPLIED
value CDATA #REQUIRED
>
<!-- If no level element is specified, then the configurator MUST not -->
<!-- touch the level of the named category. -->
<!ELEMENT category (param*,(priority|level)?,appender-ref*)>
<!ATTLIST category
class CDATA #IMPLIED
name CDATA #REQUIRED
additivity (true|false) "true"
>
<!-- If no level element is specified, then the configurator MUST not -->
<!-- touch the level of the named logger. -->
<!ELEMENT logger (level?,appender-ref*)>
<!ATTLIST logger
name CDATA #REQUIRED
additivity (true|false) "true"
>
<!ELEMENT categoryFactory (param*)>
<!ATTLIST categoryFactory
class CDATA #REQUIRED>
<!ELEMENT loggerFactory (param*)>
<!ATTLIST loggerFactory
class CDATA #REQUIRED>
<!ELEMENT appender-ref EMPTY>
<!ATTLIST appender-ref
ref CDATA #REQUIRED
>
<!-- plugins must have a name and class and can have optional parameters -->
<!ELEMENT plugin (param*, connectionSource?)>
<!ATTLIST plugin
name CDATA #REQUIRED
class CDATA #REQUIRED
>
<!ELEMENT connectionSource (dataSource?, param*)>
<!ATTLIST connectionSource
class CDATA #REQUIRED
>
<!ELEMENT dataSource (param*)>
<!ATTLIST dataSource
class CDATA #REQUIRED
>
<!ELEMENT triggeringPolicy ((param|filter)*)>
<!ATTLIST triggeringPolicy
name CDATA #IMPLIED
class CDATA #REQUIRED
>
<!ELEMENT rollingPolicy (param*)>
<!ATTLIST rollingPolicy
name CDATA #IMPLIED
class CDATA #REQUIRED
>
<!-- If no priority element is specified, then the configurator MUST not -->
<!-- touch the priority of root. -->
<!-- The root category always exists and cannot be subclassed. -->
<!ELEMENT root (param*, (priority|level)?, appender-ref*)>
<!-- ================================================================== -->
<!-- A logging event -->
<!-- ================================================================== -->
<!ELEMENT log4j:eventSet (log4j:event*)>
<!ATTLIST log4j:eventSet
xmlns:log4j CDATA #FIXED "http://jakarta.apache.org/log4j/"
version (1.1|1.2) "1.2"
includesLocationInfo (true|false) "true"
>
<!ELEMENT log4j:event (log4j:message, log4j:NDC?, log4j:throwable?,
log4j:locationInfo?, log4j:properties?) >
<!-- The timestamp format is application dependent. -->
<!ATTLIST log4j:event
logger CDATA #REQUIRED
level CDATA #REQUIRED
thread CDATA #REQUIRED
timestamp CDATA #REQUIRED
time CDATA #IMPLIED
>
<!ELEMENT log4j:message (#PCDATA)>
<!ELEMENT log4j:NDC (#PCDATA)>
<!ELEMENT log4j:throwable (#PCDATA)>
<!ELEMENT log4j:locationInfo EMPTY>
<!ATTLIST log4j:locationInfo
class CDATA #REQUIRED
method CDATA #REQUIRED
file CDATA #REQUIRED
line CDATA #REQUIRED
>
<!ELEMENT log4j:properties (log4j:data*)>
<!ELEMENT log4j:data EMPTY>
<!ATTLIST log4j:data
name CDATA #REQUIRED
value CDATA #REQUIRED
>
Static Constants for Property Types
The DataType property can have one of the values in the following table.
Name | Value | Description |
|---|---|---|
BINARY | 1 | Specifies a binary data type. Represents binary data by using an array of unsigned 8-bit bytes. |
BOOLEAN | 2 | Specifies a Boolean data type. Represents Boolean data having a value of true or false. |
DATE | 3 | Specifies a DateTime data type. Represents an instance in time as a date and time of day in accordance with ISO 8601. |
DOUBLE | 4 | Specifies a double (Float64) data type. Represents an IEEE-standard 64-bit floating-point number, which has a value ranging from -1.79769313486232e308 to +1.79769313486232e308. |
GUID | 5 | Specifies a GUID (ID) data type. Represents a Globally Unique Identifier (GUID) or DCE Universally Unique Identifier (UUID), which is a unique 128-bit number, as a string of 32 hexadecimal characters enclosed by brackets in the following format: {XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}. For example, {3F2504E0-4F89-11D3-9A0C-0305E82C3301}. |
LONG | 6 | Specifies an integer data type. Represents a signed 32-bit integer, which has a value ranging from -2,147, 483,648 to +2,147,483,647. |
OBJECT | 7 | Specifies an object data type. Represents an object that is instantiated from a Content Engine class. |
STRING | 8 | Specifies a string data type. Represents text consisting of a sequential collection of 16-bit Unicode characters. |
The Cardinality property can have one of the values in the following table.
Name | Value | Description |
|---|---|---|
ENUM | 1 | Specifies a property with enumeration cardinality. A property with enumeration cardinality is an object-valued property that returns a set collection. A set collection is a read-only collection of unique, unordered, independent objects that must be traversed sequentially. You can iterate through the items of a set collection one page of elements at a time from the server to your client application. However, if the collection changes on the server while you are iterating through it, the number, order, and values of the items in your client copy can change, even if you maintain the same reference to it. A set collection cannot hold any items other than independent objects. By contrast, a list collection can hold items of any data type, with the exception of independent objects. |
LIST | 2 | Specifies a property with list cardinality. A property with list cardinality returns a list collection. A list collection is a collection of ordered items that can either be modifiable (allowing items to be inserted, replaced, or deleted) or read-only. These items need not be unique and can be traversed in any order. When you access a list collection from the server, a complete copy of it is created on your client application, which you can iterate through one element at a time. The items in a list collection must all be of the same data type and must match the data type of the property that returns it. If the property returning a list collection is an object-valued property, all of the objects in the list collection must be dependent objects. A list collection can hold items of any data type (provided each item is of the same data type. However, if a list collection holds objects, they must all be dependent objects; only a set collection can hold independent objects. You cannot create a custom property with list cardinality. |
SINGLE | 0 | Specifies a property with single cardinality. A property with single cardinality returns a single value of the data type that the property can hold. |
Event Setup
Events recorded example query:
Events should be set up using the acce web application for the following classes.
Folder Class
File Event
Update Security Event
Unfile Event
Update Event
Deletion Event
A window displays the details of the new audit definition. The display name, event, recording level, audit type, filter expression, property name, and options with OK and cancel buttons.
Figure 4-8
The Folder Class, Audit Definitions tab is selected to add events using New
The acce web application (http://ecmukdemo6:9080/acce/) is used to set the Source (OS2) Folder Class Event records which then can be used to identify changes to the Object Store Folder objects, used by the Replication program.
The New command button is used to create a new Event definition, for example, the Unfile Event to remove a Folder Object from another Folder.
A screenshot displays a table that lists the display name with 5 files, event, enabled, subclass, audit type, and recording level under the audit information option.
Figure 4-9
The Folder Class Unfile Event
A screenshot displays the details of the new audit definition, such as the display name, event, recording level, audit type, filter expression, property name, and options with OK and cancels buttons.
Figure 4-10
The Folder Events set for Audit Definitions
Note
Don’t forget to click the Save command button or nothing will be preserved!
Document Class
Update Event
Checkin Event
Deletion Event
Update Security Event
The acce web application (http://ecmukdemo6:9080/acce/) is used to set the Source (OS2) Document Class Event records which then can be used to identify changes to the Object Store Document objects, used by the Replication program.
The New command button is used to create a new Event definition, for example, the Update Security Event to record changes to a Document Object Security.
A screenshot displays a table with 7 columns listing 11 display names with information about their event, enabled, subclass, audit type, and recording level.
Figure 4-11
The Audit Definition Event for the Document Update Security Event
A screenshot depicts objects stored under the level 2 A 1 folder. The folder has a single file titled, Level 2 A 1 document.
Figure 4-12
The Document Events set for Audit Definitions
Note
Don’t forget to click the Save command button or nothing will be preserved!
Unit Test Phases
Phase 1
- a)
Source Object Store Connection is successful.
- b)
Target Object Store Connection is successful.
- c)
C) All Test folders under a source Object root Folder, Replicate Test 1, are created in the Target Object Store irrespective of the number of overlapping runs.
Console Log output of the first Phase 1 run:
log4j:ERROR No appender named [ rfa_threads ] could be found.
log4j:ERROR No appender named [rfa2] could be found.
CE Replicate Version 3.6
Configuration property file search order:
user.dir = /root/eclipse-workspace/AUDIT_CEReplicate
user.home = /root
java.home = /opt/ibm/java-x86_64-80/jre
Loading configuration resource: FileNetBuild.properties
Configuration for:Config.AutoRefreshEnabled value:null applied
Configuration for:com.filenet.engine.LoggerRefreshInterval value:null applied
[Perf Log] No interval found. Auditor disabled.
Configuration for:FileNet.EJB.ContextProperties value:null applied
Configuration for:com.filenet.AppServerType value:null applied
Configuration for:com.filenet.AppServer value:null applied
Configuration for:com.filenet.api.useLocalAuthentication value:null applied
Configuration for:com.filenet.api.crcl.logging.chatty value:null applied
Configuration for:FileNet.crcl.implementation.api.location value:null applied
Configuration for:FileNet.crcl.implementation.api.locations value:null applied
Configuration for:FileNet.crcl.implementation.api.urls value:null applied
Configuration for:com.filenet.api.util.Id.SequentialDBType value:null applied
Configuration for:CheetahBCMode value:false applied
Configuration for:com.filenet.engine.LogRequestWithError value:null applied
Configuration for:com.filenet.engine.init.DefaultLocale value:en-us applied
Configuration for:com.filenet.locale.equivalents value:|,he_*,iw_*,|,nb_NO,nb_*,no_*,| applied
Configuration for:FileNet.WSI.HttpCookieMaxAge value:null applied
Configuration for:MaximumRetry value:null applied
Configuration for:ExpireSessionInterval value:null applied
Configuration for:BuildVersion value:dap553.1500 applied
Configuration for:FileNet.WSI.HttpChunkSize value:null applied
Configuration for:FileNet.WSI.custom.credential.class value:null applied
Configuration for:FileNet.WSI.custom.credential.usermethod value:getUserName applied
Configuration for:FileNet.WSI.custom.credential.passwordmethod value:getPassword applied
Configuration for:FileNet.WSI.TransportConnectionTimeout value:null applied
Configuration for:FileNet.WSI.stax.XMLOutputFactory value:com.ibm.xml.xlxp.api.stax.XMLOutputFactoryImpl applied
Configuration for:WSITypeAttributeRequired value:null applied
Configuration for:WSIDefaultNamespacePermitted value:null applied
Configuration for:FileNet.WSI.AutoDetectLTPAToken value:null applied
Configuration for:FileNet.WSI.stax.XMLInputFactory value:com.ibm.xml.xlxp.api.stax.XMLInputFactoryImpl applied
Begin Import Folders ...
Got ObjectStore OS2
Got ObjectStore OS1
SQL: SELECT TOP 500 f.foldername, f.Id, f.Creator, f.DateCreated, f.LastModifier, f.DateLastModified, f.Name, f.Owner, f.LockToken, f.LockTimeout, f.LockOwner, f.PathName, f.IndexationId, f.CmIndexingFailureCode, f.CmRetentionDate, f.ContainerType, f.InheritParentPermissions, f.IsHiddenContainer FROM Folder AS f WITH INCLUDESUBCLASSES WHERE f.DateCreated > 20220530T080541Z ORDER BY f.DateCreated
Configuration for:FileNet.WSI.DeserializationBufferSize value:null applied
Configuration for:FileNet.WSI.SpillCutover value:null applied
IGNORED: New Folder: Audit Master found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
IGNORED: New Folder: Case Types found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types Should be linked to :/AUDIT_TEST
IGNORED: New Folder: AUD_AuditDepartment found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment Should be linked to :/AUDIT_TEST
IGNORED: New Folder: Cases found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases Should be linked to :/AUDIT_TEST
IGNORED: New Folder: CaseStructure found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/CaseStructure Should be linked to :/AUDIT_TEST
Realm Name: o=sample
IGNORED: New Folder: 2022 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 06 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 09 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0044 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09/0044 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000100001 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09/0044/000000100001 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 11 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/11 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0234 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/11/0234 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000110001 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/11/0234/000000110001 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 18 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0118 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0118 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120001 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0118/000000120001 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0101 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0101 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120002 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0101/000000120002 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0042 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0042 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120003 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0042/000000120003 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0141 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0141 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120004 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0141/000000120004 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0250 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0250 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120005 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0250/000000120005 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0156 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0156 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120006 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/18/0156/000000120006 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 19 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/19 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0248 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/19/0248 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000120007 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/19/0248/000000120007 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 07 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 07 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 0279 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07/0279 Should be linked to :/AUDIT_TEST
IGNORED: New Folder: 000000130001 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07/0279/000000130001 Should be linked to :/AUDIT_TEST
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Import 16 Folders took 6043 milliseconds
Begin Delete Docs ...
SQL: SELECT TOP 500 e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId FROM Event AS e WITH INCLUDESUBCLASSES INNER JOIN CLASSDEFINITION AS cd WITH INCLUDESUBCLASSES ON e.SourceClassId = cd.Id WHERE (e.DateCreated >= 20220530T080541Z) AND (cd.SymbolicName <>'Folder') AND (cd.SymbolicName <>'FOLDER_ATTRIBUTE') AND (e.EventStatus = 0) ORDER BY e.CmAuditSequence,e.DateCreated
Delete 10 Docs took 6933 milliseconds
Begin Import Docs ...
Start Search Date : 20220530T080541Z
Start Delta Hours : -1
SQL: SELECT TOP 500 d.* FROM Document AS d WITH INCLUDESUBCLASSES WHERE d.DateLastModified > 20220530T080541Z and d.IsCurrentVersion = true and d.IsReserved = false ORDER BY d.DateLastModified,d.MajorVersionNumber, d.MinorVersionNumber
IGNORED: New Document: PageResources.zip found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
IGNORED: New Document: ViewResources.zip found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
IGNORED: New Document: AC.pdf found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09/0044/000000100001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: AuditEventHandler found linked to external Folder Path : /CodeModules Should be linked to :/AUDIT_TEST
IGNORED: New Document: Test of the Audit Event found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09/0044/000000100001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: Test Audit Events found linked to external Folder Path : /Published Should be linked to :/AUDIT_TEST
IGNORED: New Document: Audit Event Test Version 2 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/09/0044/000000100001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: Automatic Folder ID added found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/06/11/0234/000000110001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: Solution Workflow Collection not linked to any Folder Path !
IGNORED: New Document: Detail Deployment Log found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
IGNORED: New Document: Error Deployment Log found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
IGNORED: New Document: AUDOperations.jar found linked to external Folder Path : /CodeModules Should be linked to :/AUDIT_TEST
IGNORED: New Document: Solution Workflow Collection not linked to any Folder Path !
IGNORED: New Document: AUDOperations.jar found linked to external Folder Path : /CodeModules Should be linked to :/AUDIT_TEST
IGNORED: New Document: JMS Message Trigger found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07/0279/000000130001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: Second Audit Report Example 08-July-2022 found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07/0279/000000130001 Should be linked to :/AUDIT_TEST
IGNORED: New Document: AUD_JMSMessageWorkflow found linked to external Folder Path : /Published Should be linked to :/AUDIT_TEST
IGNORED: New Document: AuditReport00005_Engineering.pdf found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master/Case Types/AUD_AuditDepartment/Cases/2022/07/07/0279/000000130001 Should be linked to :/AUDIT_TEST
Realm Name: o=sample
IGNORED: New Document: Resources.js not linked to any Folder Path !
IGNORED: New Document: Solution Definition not linked to any Folder Path !
IGNORED: New Document: PE Configuration not linked to any Folder Path !
IGNORED: New Document: General Claim Workflow Definition not linked to any Folder Path !
IGNORED: New Document: Solution Definition not linked to any Folder Path !
IGNORED: New Document: PE Configuration not linked to any Folder Path !
IGNORED: New Document: Audit Department Workflow Definition not linked to any Folder Path !
Realm Name: o=sample
Realm Name: o=sample
Realm Name: o=sample
Import 13 Docs took 13420 milliseconds
Begin Delete Folders ...
SQL: SELECT TOP 500 e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.SourceObject,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId FROM Event AS e WITH INCLUDESUBCLASSES INNER JOIN CLASSDEFINITION AS cd WITH INCLUDESUBCLASSES ON e.SourceClassId = cd.Id WHERE (e.DateCreated > 20220530T080541Z) AND ((cd.SymbolicName = 'Folder') OR (cd.SymbolicName = 'FOLDER_ATTRIBUTE') ) AND (e.EventStatus = 0) ORDER BY e.CmAuditSequence,e.DateCreated
Delete 0 Folders took 838 milliseconds
Begin Update Docs ...
SQL: SELECT TOP 500 e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId FROM Event AS e WITH INCLUDESUBCLASSES INNER JOIN CLASSDEFINITION AS cd WITH INCLUDESUBCLASSES ON e.SourceClassId = cd.Id WHERE (e.DateCreated > 20220530T080541Z) AND (cd.SymbolicName <>'Folder') AND (cd.SymbolicName <>'FOLDER_ATTRIBUTE') AND (e.EventStatus = 0) ORDER BY e.CmAuditSequence,e.DateCreated
Update 242 Docs took 1591 milliseconds~
The OS1 replicate /AUDIT_TEST folder Object Store now has the replicated folder structure and linked documents exported from the OS2 Object Store source.
A screenshot depicts objects stored under the level 2 A 1 folder. The folder has a single file titled, Level 2 A 1 document.
Figure 4-12A
The original OS2 test data folder and document structure to be replicated
A screenshot of the command base with the use of applibs logs, and applibs jar logs.
Figure 4-12B
The OS1 test data folder and document structure from the OS2 original
Notice that the exported OS1, AUDIT_TEST folder document, Date Created property entry (July 31, 2022 at 3:58:37 PM PDT), is exactly the same as from the original document in the OS2 Object Store, as is required.
Phase 2
- a)
Ensure matching property values in the Source Object and with the same security ACLs and GUID values.
- b)
Ensure major/minor versions of documents under the source Object Folders, Replicate Test 1 and subfolders, are created and linked correctly in the Target Object.
- c)
Ensure Folders and Documents created in other Folder paths outside the defined path are not copied.
Note
This was seen in the preceding output log, for example:
IGNORED: New Folder: Audit Master found linked to external Folder Path : /IBM Case Manager/Solution Deployments/Audit Master Should be linked to :/AUDIT_TEST
- d)
Add new code to test for an existing document in the Target Object Store and delete and recreate it.
Phase 3
- a)
First replication – Verify the object store and folders.
The same object store and folders are found at the target.
- b)
Remove a document at the source – The same document is removed at the target.
- c)
Verify the folder change – Amend the name of the existing folder at the source.
The change is synced to the target.
- d)
Remove a folder at the source – The same folder is removed at the target.
Phase 4
- a)
Complete Deployment Manager full replication Test.
Update to encrypt the Alan password – Automatic config.xml updates.
(This is in the code listing, tested and working!)
Update to automatically update the next replication start date in the config.xml file.
(This was checked in the config.xml file and updated as expected.)
- b)
Release Batch Replication.sh and Java program for testing.
- c)
Create the documentation (this chapter).
Creating the Java Projects
The Eclipse Java IDE program used in Chapters 2 and 3 was used in this chapter to create a set of projects for the development of the Java Replication program .jar files with the outline settings as shown in Table 4-1.
Table 4-1
The Java Eclipse Projects used for support of the Replication program
Project Name | Library .Jar Name | Source Java File(s) | Brief Description |
|---|---|---|---|
FileNetConnection | fn_connection.jar | CEConnection.java | Loads the user and password |
FN_Connect | fn_connect.jar | CEConnect.java | Uses the FileNet Web Service Interface to connect (using the FileNetP8WSI stanza) |
AUDIT_CEReplicate | replication.jar | CEReplicate.java | The main Replication program in the package com.ibm.ce.rep.utils |
AUDIT_Common | fn_common.jar | CEReplicateConfig.java CEMigrateConfig.java | Reads in the XML configuration parameters for the Replication |
AUDIT_Utils | fn_utils.jar | PropsUtil.java | Loads configuration details from a properties file |
Part 3 – Code Developed for the Replication Program
CEReplicate – 2957 Lines of Java Code – replication.jar
The supporting compiled jar files and standard encryption .jars are copied into the /opt/replication/libs/applibs subfolder.
A screenshot of the command base under forward slash opt forward slash replication with the use of applibs jars libs for the creation of five jars subdirectories.
Figure 4-12C
The applibs, jars, and libs subdirectories are built under /opt/replication
A command under forward slash replication dot s h includes classpath and error data as no appender name could be found.
Figure 4-12D
The /opt/replication/libs/applibs subfolder .jar files
(base) [root@ECMUKDEMO6 libs]# ls
applibs other
(base) [root@ECMUKDEMO6 libs]# pwd
/opt/replication/libs
(base) [root@ECMUKDEMO6 libs]# cd ..
(base) [root@ECMUKDEMO6 replication]# ls
applibs jars libs logs
(base) [root@ECMUKDEMO6 replication]# cp /root/eclipse-workspace/AUDIT_CEReplicate/src/log4j.dtd .
(base) [root@ECMUKDEMO6 replication]# cp /root/eclipse-workspace/AUDIT_CEReplicate/src/log4j.xml .
(base) [root@ECMUKDEMO6 replication]# ls
applibs jars libs log4j.dtd log4j.xml logs
(base) [root@ECMUKDEMO6 replication]# chmod 775 log4j.*
(base) [root@ECMUKDEMO6 replication]# ls
applibs jars libs log4j.dtd log4j.xml logs
(base) [root@ECMUKDEMO6 replication]# cp /root/eclipse/java-2021-12/eclipse/fn_*.jar .
(base) [root@ECMUKDEMO6 replication]# ls
applibs fn_connection.jar fn_eventhandlers.jar jars log4j.dtd logs
fn_common.jar fn_connect.jar fn_utils.jar libs log4j.xml
(base) [root@ECMUKDEMO6 replication]# cd applibs
(base) [root@ECMUKDEMO6 applibs]# mv ../fn*.jar .
(base) [root@ECMUKDEMO6 applibs]# ls
fn_common.jar fn_connection.jar fn_connect.jar fn_eventhandlers.jar fn_utils.jar
The latest release of the apache codec jar can be downloaded from the following URL:
https://commons.apache.org/proper/commons-codec/download_codec.cgi
A Xerces Java library file can be downloaded from the zip file:
http://archive.apache.org/dist/xml/xerces-j/Xerces-J-bin.1.4.4.zip
(An import used in CEMigrateConfig.java code for XML parsing)
A screenshot depicts 24 lines of command under forward slash opt forward slash replication with C E replicate version 3.6 and configuration file search order and highlighted console dot log at the end.
Figure 4-12E
The first run of the Replication.sh shell script (the CLASSPATH is echoed)
A screenshot depicts the command under forward slash opt forward slash replication with the ignored files, highlighted replication, imported 13 documents, and updated 242 documents in middle.
Figure 4-12F
The console.log file from the first run of the Replication.sh shell script
The end of the console.log shows the successful transfer of the folders and documents.
A screenshot depicts the command under forward slash opt forward slash replication with c e login import, file name, and folder subclass highlighted as 20220801 T 1516262.
Figure 4-12G
The Replication program run statistics of the console.log file is shown
A command under audit imports audit documents dots log, 5 lines of data, 4 finished times, documents that failed to be processed, and a cursor at the end.
Figure 4-12H
The Replication program has successfully updated the start search date
A command includes applibs logs x m l, dot jar, and functions of event handlers to present the details of imported documents.
Figure 4-12I
The AuditImportAuditDocs.log shows the details of imported documents
An activity titled box depicts the options under file as new, further leading to the project.
Figure 4-13
The setup of the supporting file systems for the Replication program
Project Creation
The Eclipse New project, AUDIT_CEReplicate, is created.
A new project titled box depicts the 4 types of wizards with java project under java selected.
Figure 4-14
The File ➤ New ➤ Project menu option is selected for the New Project wizard
The Java Project type is selected.
A screenshot depicts the inputs of the user's new project name, file location, J R E project, project layout with default configuration, add project sets, with cancel and finish options.
Figure 4-15
The Java Project is selected with the Eclipse Project wizard
A number of separate projects will be created for each custom utility .jar file we will reference.
A screenshot of a new java project box depicts a folder under the Java server setting. A project AUDIT underscore C E Replicate is listed with s r c folder into it.
Figure 4-16
The Project Name, AUDIT_CEReplicate, is entered and the 1.8 JRE selected
A popup window, open associate perspective? illustrates the perspective with no and open perspective options at the bottom right.
Figure 4-17
The parameters are set for the Java project, and Finish is clicked
A screenshot depicts 2 folders under a package explorer. The folder titled, s r c under Audit underscore C E replicate is selected and its J units run and errors are shown.
Figure 4-18
The Open Perspective command button is clicked for Java development
A screenshot depicts the way to create a new package under s r c, listed under the folder Audit underscore C E replicate.
Figure 4-19
The AUDIT_CEReplicate Project is now available for Java packages and code
A new Java package is created for the CEReplicate class.
A dialog box titled new Java packages with options to create new ones, source folder named Audit underscore C E Replicate, the name, com dot I B M dot c e dot rep dot utils, is highlighted.
Figure 4-20
The right-mouse click is used to select the New ➤ Package menu option
A screenshot depicts the way to create a new class under com dot I B M dot c e dot rep dot utils, is listed under the folder Audit underscore C E replicate.
Figure 4-21
The Java package name, com.ibm.ce.rep.utils, is created by clicking Finish
A dialog box titled a new Java class has a source folder, packages, name C E replicate highlighted, modifiers set to public, superclass, interfaces, method stubs, and highlight finish option.
Figure 4-22
The CEReplicate Class is added to the com.ibm.ce.rep.utils package
A screenshot of C E replicates dot java and starts listing 15 programming codes by I B M along with decryption java classes.
Figure 4-23
The CEReplicate class name is entered and the Finish command clicked
Adding the Base CEReplicate Java Code
The base CEReplicate.java code is pasted in from the standard code module as a starting point for the development and comments and new Java code added for changes for our functionality.
A screenshot displays Audit C E R properties for replicating the Java build path, including resource files and builders. It highlights the Java build path and adds an external J A R, the right side.
Figure 4-24
The base Java Code is added for CEReplicate.java
Adding the IBM FileNet Libraries to the AUDIT_CEReplicate Classpath
IBM FileNet P8 5.5.x standard libraries are added to the AUDIT_CEReplicate Project Classpath. This should correct the flagged issues on the missing imports from the standard IBM FileNet API jace.jar and its supporting library .jar files.
A screenshot displays the J A R selection tab that lists 7 folders under file net jars. 4 folders from the bottom are selected.
Figure 4-25
IBM FileNet P8 .jar libraries are added to the Classpath
A screenshot displays the J A R selection tab that lists 4 projects under the file net jars library. All the files are selected.
Figure 4-26
The Apply and Close command button is clicked to add the library .jar files
The other projects are similarly created as described in Table 4-1 (with the code copied from the Java code listings in the following subsections).
Adding the Apache Commons Codec 1.5 Library
A screenshot displays Apache commons C o d e c 1.15 lists information about binaries, sources and archives.
Figure 4-27
The latest Commons Codec Version 1.5 is downloaded from the Apache site
The URL https://commons.apache.org/proper/commons-code.codec.cgi is entered in the Firefox browser, as shown in Figure 4-27, to download the commons-codec-1.15-bin.tar.gz file for our Linux Eclipse project.
A popup of download titled, opening common codecs 1.15 bin dot tar dot g z enquires either to open with or save the file by Mozilla Firefox. The save option is highlighted.
Figure 4-28
The downloaded commons-codec-1.15-bin.tar.gz file is saved
The downloaded commons-codec-1.15-bin.tar.gz file is saved to the /root/Downloads browser folder.
A screenshot displays 4 zip folders under the download tab. The zip folder titled, common codecs 1.15 bin dot tar dot g z is highlighted.
Figure 4-29
The downloaded commons-codec-1.15-bin.tar.gz file is unpacked
The downloaded commons-codec-1.15-bin.tar.gz file is unpacked using the Linux File Explorer desktop utility by double-clicking the .gz file to view the packed files.
A screenshot displays 6 folders and 3 text files under the path name common codecs 1.15. The folder titled, common codecs 1.15 dot jar is selected.
Figure 4-30
The unpacked .gz file displays the commons-codec-1.15.jar file
The commons-codec-1.15.jar file is now copied to the /opt/FileNetJars folder.
A screenshot displays options to choose a folder to copy from 8 dot jar extension folders.
Figure 4-31
The /opt/FileNetJars folder is browsed to copy the commons-codec-1.15.jar
Adding the Supporting External Library JAR Files
The properties for the AUDIT_CEReplicate project show the Java Build Path, and we can use the Add External JARs command button to browse and select the highlighted commons-code-1.15.jar file from the /opt/FileNetJars folder path.
A screenshot displays the properties for audit underscore C E replicate tab that lists 5 projects under the file net jars library. A project at the top is selected.
Figure 4-32
The Add External JARs command button is used to load the JAR files required
The commons-code-1.15.jar file we downloaded earlier is used for the password encryption in the config.xml file.
A screenshot displays the J A R selection tab that lists 8 folders under the J A R selection tab. The folder at the bottom titled, common codecs 1.15 dot jar is selected.
Figure 4-33
The commons-codec-1.15.jar is selected from the /opt/FileNetJars folder
The Eclipse properties Java Build Path for the AUDIT_CEReplicate project is updated to include the commons-codec-1.15.jar file which is missing from the import statement, highlighted in Figure 4-34, it can be seen that the addition of the Jar file has fixed the flagged issue of the missing Base64 class.
A screenshot of a tab titled, C E replicate dot java lists import functions. The function at line 60, dot C E connection is highlighted.
Figure 4-34
The import org.apache.commons.codec.binary.Base64 class is now fixed
Adding the Xerces Java Library for XML File Processing
A popup of download titled, opening Xerces J bin dot 1.4.4 dot zip f enquires either to open with or save the file by Mozilla Firefox. The save option is highlighted.
Figure 4-35
The Xerces-J-bin.1.4.4.zip file is downloaded from the Apache website
The Xerces-J-bin.1.4.4.zip file is downloaded from the https://archive.apache.org/dist/xml/xerces-j/Xerces-J-bin.1.4.4.zip Apache jar download site.
A screenshot displays 4 folders under the download. A folder titled Xerces J bin dot 1.4.4 dot zip at the top is highlighted.
Figure 4-36
The Xerces-J-bin.1.4.4.zip file is shown in the /root/Downloads Linux folder
The Xerces-J-bin.1.4.4.zip file is downloaded and unpacked using the Extract function of the Linux File Explorer desktop utility.
A screenshot displays 5 folders and 2 files under path Xerces 1 underscore 4 underscore 4. A folder titled Xerces dot jar is highlighted.
Figure 4-37
The xerces.jar file is extracted from the Xerces-J-bin.1.4.4.zip file
The unpacked xerces.jar file is extracted to the /opt/FileNetJars folder location.
A screenshot displays 9 folders to extract under the file net jars.
Figure 4-38
The unpacked xerces.jar file is extracted to the /opt/FileNetJars folder
The xerces.jar file is used to satisfy the XML parsing requirements for the Java replication program.
A screenshot displays the J A R selection tab that lists 9 projects under the file net jars library. The project at the top titled, Xerces dot jar is highlighted.
Figure 4-39
The unpacked xerces.jar file is added to the Java Build Path
The properties of the AUDIT_CEReplicate project are selected to add the xerces.jar file to the Java Build Path property.
A screenshot displays a tab with selected Java build path that lists 9 projects under the file net jars library. The project at the top titled, Xerces dot jar is highlighted.
Figure 4-40
The xerces.jar is now added to the Java Build Path
The import statement needed to be updated to adjust the package path of the OutputFormat class as shown in Figure 4-41.
An illustration of double slash A S B development limited Apache class on 2022, July 29, org dot Apache X M L, and import org dot Apache dot X M L dot serialize dot output format semicolon.
Figure 4-41
The adjusted import statement for the OutputFormat class
Adding the Supporting Java Projects
The four supporting projects, FileNetConnection, FN_Connect, AUDIT_Common, and AUDIT_Utils, are run as follows to create the supporting .jar files (as defined in Table 4-1).
In the AUDIT_CEReplicate project source folder, we first create a config subfolder.
A dialog box titled, new folder displays the option to create a new folder resource under the selected parent folder and desired folder name with an option to cancel or finish.
Figure 4-42
The Config folder is created for the config.xml file
The config.xml file is created to hold the Replication program parameters for the source and target Object Store names, the administrator name and encrypted password for the IBM FileNet content engine system account, and the included users and group security for access to the replicated folders and documents.
A dialog box titled, create new file displays the option to create a new file resource under the selected parent folder config and desired folder name with an option to cancel or finish.
Figure 4-43
The Config.xml file is created – note later changed to config.xml (lowercase)
The supporting project jar files are generated as shown in Table 4-1.
A dialog box titled, activities illustrates an option to export audit underscore export listed under package explorer.
Figure 4-44
The AUDIT_Common project is selected to export the fn_common.jar file
A dialog box titled, export displays the option to select an export wizard with a jar file under the Java folder selected with an option to cancel or next.
Figure 4-45
The Java JAR file export option is selected
A dialog box titled, J A R export displays the option to select the resources to export from the list and the destination path with an option to compress, back, finish, cancel or next.
Figure 4-46
The fn_common.jar export path is selected
A dialog box titled, J A R export displays options to select for handling problems, and a description file path with an option to compress, back, finish, cancel or next. The next option is highlighted.
Figure 4-47
The AUDIT_Common.jardesc file is selected for build, to use for faster rebuilds
A dialog box titled, J A R export displays options to specify the manifest, manifest file path, seal content options and options to compress, back, finish, and cancel.
Figure 4-48
The jar MANIFEST.MF file is selected for creation and Finish clicked
A warning box illustrates containing J A R file exports with compile warnings with an option to details and ok.
Figure 4-49
The Details command button displays the compile warnings found
The Java JAR Build XML, AUDIT_Common.jardesc, is as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<jardesc>
<jar path="/root/eclipse/java-2021-12/eclipse/fn_common.jar"/>
<options buildIfNeeded="true" compress="true" descriptionLocation="/AUDIT_Common/src/AUDIT_Common.jardesc" exportErrors="true" exportWarnings="true" includeDirectoryEntries="false" overwrite="false" saveDescription="true" storeRefactorings="false" useSourceFolders="false"/>
<storedRefactorings deprecationInfo="true" structuralOnly="false"/>
<selectedProjects/>
<manifest generateManifest="true" manifestLocation="/AUDIT_Common/bin/MANIFEST.MF" manifestVersion="1.0" reuseManifest="false" saveManifest="true" usesManifest="true">
<sealing sealJar="false">
<packagesToSeal/>
<packagesToUnSeal/>
</sealing>
</manifest>
<selectedElements exportClassFiles="true" exportJavaFiles="false" exportOutputFolder="false">
<file path="/AUDIT_Common/.classpath"/>
<file path="/AUDIT_Common/.project"/>
<javaElement handleIdentifier="=AUDIT_Common/src"/>
</selectedElements>
</jardesc>
A dialog box titled, export displays the option to select the resources to export, an export destination of fn underscores utils dot jar with an option to compress, back, next, cancel or finish.
Figure 4-50
The fn_utils.jar export path is selected
The Java JAR Build XML, AUDIT_Utils.jardesc, for the fn_utils.jar is as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<jardesc>
<jar path="/root/eclipse/java-2021-12/eclipse/fn_utils.jar"/>
<options buildIfNeeded="true" compress="true" descriptionLocation="/AUDIT_Utils/src/AUDIT_Utils.jardesc" exportErrors="true" exportWarnings="true" includeDirectoryEntries="false" overwrite="false" saveDescription="true" storeRefactorings="false" useSourceFolders="false"/>
<storedRefactorings deprecationInfo="true" structuralOnly="false"/>
<selectedProjects/>
<manifest generateManifest="true" manifestLocation="/AUDIT_Utils/bin/MANIFEST.MF" manifestVersion="1.0" reuseManifest="false" saveManifest="true" usesManifest="true">
<sealing sealJar="false">
<packagesToSeal/>
<packagesToUnSeal/>
</sealing>
</manifest>
<selectedElements exportClassFiles="true" exportJavaFiles="false" exportOutputFolder="false">
<file path="/AUDIT_Utils/.classpath"/>
<file path="/AUDIT_Utils/.project"/>
<javaElement handleIdentifier="=AUDIT_Utils/src"/>
</selectedElements>
</jardesc>
A dialog box titled, export displays the option to select the resources to export, an export destination of fn underscores connect dot jar with an option to compress, back, next, cancel or finish.
Figure 4-51
The fn_connect.jar export path is selected
The Java JAR Build XML, FN_Connect.jardesc, for the fn_connect.jar is as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<jardesc>
<jar path="/root/eclipse/java-2021-12/eclipse/fn_connect.jar"/>
<options buildIfNeeded="true" compress="true" descriptionLocation="/FN_Connect/src/FN_Connect.jardesc" exportErrors="true" exportWarnings="true" includeDirectoryEntries="false" overwrite="false" saveDescription="true" storeRefactorings="false" useSourceFolders="false"/>
<storedRefactorings deprecationInfo="true" structuralOnly="false"/>
<selectedProjects/>
<manifest generateManifest="true" manifestLocation="/FN_Connect/bin/MANIFEST.MF" manifestVersion="1.0" reuseManifest="false" saveManifest="true" usesManifest="true">
<sealing sealJar="false">
<packagesToSeal/>
<packagesToUnSeal/>
</sealing>
</manifest>
<selectedElements exportClassFiles="true" exportJavaFiles="false" exportOutputFolder="false">
<file path="/FN_Connect/.project"/>
<javaElement handleIdentifier="=FN_Connect/src"/>
<file path="/FN_Connect/.classpath"/>
</selectedElements>
</jardesc>
A screenshot displays the J A R selection tab that lists 9 projects under the path eclipse. The project titled, fn underscores connect dot jar is highlighted.
Figure 4-52
The generated fn_connect.jar supporting jar file for the Replication program
The supporting jar file fn_connect.jar is added to the Java Build Path of the FileNet fn_connection.jar creation project FileNetConnection.
A screenshot displays 9 jar projects listed under Java build path and an option to add external jars. The project titled, an eclipse is highlighted.
Figure 4-53
The generated fn_connect.jar is added to the FileNetConnection project
A screenshot displays the J A R selection tab that lists 9 projects under the file net jars library. The project at the top titled, fn underscore common dot jar is highlighted.
Figure 4-54
The generated fn_common.jar file is selected from the eclipse root path
A screenshot displays 9 jar projects listed under Java build path and option to add external jars. 2 projects titled, fn underscore common dot jar and fn underscore connect dot jar are highlighted.
Figure 4-55
The generated fn_common.jar is added to the FileNetConnection project
A dialog box titled, export displays the option to select the resources to export, an export destination of fn underscore connection dot jar with an option to compress, back, next, cancel or finish.
Figure 4-56
The fn_connection.jar export path is selected
A dialog box titled, export displays the option to specify the manifest, manifest file, and seal contest with an option to back, cancel or finish.
Figure 4-57
The fn_connection.jar export MANIFEST.MF file path is selected
The Java JAR Build XML, FileNetConnection.jardesc, is as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<jardesc>
<jar path="/root/eclipse/java-2021-12/eclipse/fn_connection.jar"/>
<options buildIfNeeded="true" compress="true" descriptionLocation="/FileNetConnection/src/FileNetConnection.jardesc" exportErrors="true" exportWarnings="true" includeDirectoryEntries="false" overwrite="false" saveDescription="true" storeRefactorings="false" useSourceFolders="false"/>
<storedRefactorings deprecationInfo="true" structuralOnly="false"/>
<selectedProjects/>
<manifest generateManifest="true" manifestLocation="/FileNetConnection/bin/MANIFEST.MF" manifestVersion="1.0" reuseManifest="false" saveManifest="true" usesManifest="true">
<sealing sealJar="false">
<packagesToSeal/>
<packagesToUnSeal/>
</sealing>
</manifest>
<selectedElements exportClassFiles="true" exportJavaFiles="false" exportOutputFolder="false">
<file path="/FileNetConnection/.project"/>
<file path="/FileNetConnection/.classpath"/>
<javaElement handleIdentifier="=FileNetConnection/src"/>
</selectedElements>
</jardesc>
Updating the Java Build Path with the Required Projects
A screenshot displays the properties for audit underscore C E replicate with Java build path selected, with an option to add new under the projects folder.
Figure 4-58
The Java Build Path Projects tab is selected and the Add command clicked
In Figure 4-58, we select the AUDIT_CEReplicate, Replication.jar build project properties and click the Java Build Path menu item and then select the Projects tab to identify the Required projects on the build path.
A screenshot displays a box of required project selections that lists 7 projects. A U D I T underscores Common, Utils, file connection, function connect and O K tab is selected.
Figure 4-59
The supporting Projects selected for addition to the Eclipse Java Build Path
A screenshot displays the properties for audit underscore C E replicate with Java build path selected, with 4 selected projects under the projects folder.
Figure 4-60
The supporting Projects are added to the AUDIT_CEReplicate project
A screenshot displays the J A R selection tab that lists 10 projects under the path eclipse. The 4 projects are highlighted.
Figure 4-61
The four created supporting jar files in the Eclipse root folder on Linux
The Add External JARs command is selected, and the supporting jars displayed in Figure 4-61 are added to the AUDIT_CEReplicate project in the Java Build Path Libraries tab list.
A screenshot displays the properties for audit underscore C E replicate with Java build path selected lists 9 jar projects and options to add new external.
Figure 4-62
The four supporting .jar libraries are added to the main Project
Updating the Supporting Jar Library File Security
The output screen displays 10 lines of commands, the base, square bracket, root at E C M U K D E M 0 6 eclipses, hashtag p w d. In the third-line function, J A R In the fifth line, write c h m o d.
Figure 4-63
The security on the four generated supporting jar files is changed to allow access for the project build
The output screen displays 10 lines of commands, starting from K D E M 0 6 eclipse l s a slash root, g r r d b m with date and time, 4 and 5 lines of hashtag c h m o d and audit, and ending with a cursor.
Figure 4-64
The security on the config.xml file is updated to support the search date updates which set the next search start date for source object store changes
Updating the IBM FileNet Object Store Security
A dialog box lists available users and groups, selected users and groups, and permission factors as type, apply to, and group with 13 options.
Figure 4-65
The security on the two Object Stores OS1 and OS2 is updated for the connected Content Engine administrator user used by the Replication program
The security displayed in Figure 4-65 is used to ensure the user has the ACL security options for Modify certain system properties (also programmatically added using the API).
The output screen displays 30 lines of commands, including, in square brackets, root at E C M U K D E M 0 6, imported A U D I T folders dot log file with date and time.
Figure 4-66
The Audit log file security is changed to support the program writes
Running the Replication Program Tests
A screenshot displays the folders under O S 1. The folder AUDIT underscores TEST folder under the root folder, under browse is highlighted.
Figure 4-67
The root folder /AUDIT_TEST is created in the target Object Store, OS1
The output screen starts with import start, guide, source file, from D0891 A 81 and end with 4642 A 249 D 7 A 0 and the last 4 lines present 36 documents processed, failed 0 and finished.
Figure 4-68
The 36 Documents processed displayed in the AuditImportAuditDocs.log
A screenshot displays eclipse workspace with highlighted AUDIT underscore C E, Common, Utils, file connection, function underscore connect, Java code 40 lines with a private keyword of C E replicate.
Figure 4-69
The completed project set for the Replication.jar file build
A dialog box titled, export displays the option to select the resources to export, an export destination of replication dot jar with an option to compress, back, next, cancel or finish.
Figure 4-70
The main replication.jar file is exported for use
A dialog box titled, export displays the option to select for handling problems, description file AUDIT underscore C E Replicate dot jar with an option to back, next, cancel or finish.
Figure 4-71
The AUDIT_CEReplicate.jardesc file is created for easy rebuilds
A dialog box titled, export displays the option to specify the manifest, manifest file, and seal contest with an option to back, cancel or finish.
Figure 4-72
The MANIFEST.MF file is selected for addition to the bin folder
The Java JAR Build XML, AUDIT_CEReplicate.jardesc, is as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<jardesc>
<jar path="/root/eclipse/java-2021-12/eclipse/replication.jar"/>
<options buildIfNeeded="true" compress="true" descriptionLocation="/AUDIT_CEReplicate/src/AUDIT_CEReplicate.jardesc" exportErrors="true" exportWarnings="true" includeDirectoryEntries="false" overwrite="false" saveDescription="true" storeRefactorings="false" useSourceFolders="false"/>
<storedRefactorings deprecationInfo="true" structuralOnly="false"/>
<selectedProjects/>
<manifest generateManifest="true" manifestLocation="/AUDIT_CEReplicate/bin/MANIFEST.MF" manifestVersion="1.0" reuseManifest="false" saveManifest="true" usesManifest="true">
<sealing sealJar="false">
<packagesToSeal/>
<packagesToUnSeal/>
</sealing>
</manifest>
<selectedElements exportClassFiles="true" exportJavaFiles="false" exportOutputFolder="false">
<folder path="/AUDIT_CEReplicate/config"/>
<javaElement handleIdentifier="=AUDIT_CEReplicate/src"/>
<file path="/AUDIT_CEReplicate/.project"/>
<file path="/AUDIT_CEReplicate/.classpath"/>
</selectedElements>
</jardesc>
Result with the Exact GUID Copy of a Replicated Document
The following two screenshots demonstrate the unique Replication we have achieved where the source and target documents have exactly the same core system properties, including the Creation Date and GUID (ID) values.
The OS2 source of the replicated Document Level2A1 Document is shown in Figure 4-73.
A screenshot displays the O S 2 H T M L file in level 2 A 1 document properties lists property name, with values and highlight the I D of property.
Figure 4-73
The Document in OS2 GUID {907B5682-0000-C91D-B246-DA41D0421E00}
The OS1 target of the replicated Document Level2A1 Document is shown in Figure 4-74.
A screenshot displays the O S 2 H T M L file in level 2 A 1 document properties lists property name, with values and highlight the I D of property.
Figure 4-74
The Document in OS1 GUID {907B5682-0000-C91D-B246-DA41D0421E00}
Part 4 – Code Listing – CEReplicateConfig
The CEReplicateConfig Java code contains the Main program calling the Java methods to load the config.xml file parameters and query the source Target Object store for the changed Document and Folder objects based on the search date read in from the file.
The program then builds an exact replica of all the folders and documents under the designated “root” folder (AUDIT_TEST in the example) of the Target Object Store.
/**
IBM grants you a non-exclusive copyright license to use all programming code
examples from which you can generate similar function tailored to your own
specific needs.
All sample code is provided by IBM for illustrative purposes only.
These examples have not been thoroughly tested under all conditions. IBM,
therefore cannot guarantee or imply reliability, serviceability, or function of
these programs.
All Programs or code component contained herein are provided to you "AS IS"
without any warranties of any kind.
The implied warranties of non-infringement, merchantability and fitness for a
particular purpose are expressly disclaimed.
© Copyright IBM Corporation 2013, ALL RIGHTS RESERVED.
*/
package com.ibm.ce.rep.utils;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.InputStream;
import java.io.UnsupportedEncodingException;
import java.lang.reflect.TypeVariable;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Iterator;
//ASB ... 4 001 Encryption / Decryption Java Classes
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import java.io.UnsupportedEncodingException;
import java.security.spec.AlgorithmParameterSpec;
import java.security.spec.KeySpec;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;
//ASB ... 4 END
import com.filenet.api.admin.PropertyDefinition;
import com.filenet.api.core.BatchItemHandle;
import org.apache.commons.codec.binary.Base64;
import org.apache.log4j.Logger;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
//import com.ibm.filenet.ce.CEConnection;
import com.filenet.api.collection.AccessPermissionList;
import com.filenet.api.collection.ContentElementList;
import com.filenet.api.collection.GroupSet;
import com.filenet.api.collection.IndependentObjectSet;
import com.filenet.api.collection.PropertyDefinitionList;
import com.filenet.api.collection.StringList;
import com.filenet.api.collection.UserSet;
import com.filenet.api.collection.VersionableSet;
import com.filenet.api.constants.AccessRight;
import com.filenet.api.constants.AccessType;
import com.filenet.api.constants.AutoClassify;
import com.filenet.api.constants.AutoUniqueName;
import com.filenet.api.constants.CheckinType;
import com.filenet.api.constants.ClassNames;
import com.filenet.api.constants.CompoundDocumentState;
import com.filenet.api.constants.DefineSecurityParentage;
import com.filenet.api.constants.FilteredPropertyType;
import com.filenet.api.constants.JoinComparison;
import com.filenet.api.constants.JoinOperator;
import com.filenet.api.constants.PrincipalSearchAttribute;
import com.filenet.api.constants.PrincipalSearchType;
import com.filenet.api.constants.PropertyNames;
import com.filenet.api.constants.RefreshMode;
import com.filenet.api.constants.ReservationType;
import com.filenet.api.constants.VersionStatus;
import com.filenet.api.core.ContentTransfer;
import com.filenet.api.core.Document;
import com.filenet.api.core.EngineObject;
import com.filenet.api.core.Factory;
import com.filenet.api.core.Factory.AccessPermission;
import com.filenet.api.core.Factory.ClassDefinition;
import com.filenet.api.core.Factory.ClassDescription;
import com.filenet.api.core.Factory.Domain;
import com.filenet.api.core.Factory.DynamicReferentialContainmentRelationship;
//import com.filenet.api.core.Factory.PropertyDefinitionDateTime;
import com.filenet.api.core.Factory.Realm;
import com.filenet.api.core.Factory.User;
import com.filenet.api.core.Folder;
import com.filenet.api.core.IndependentObject;
import com.filenet.api.core.IndependentlyPersistableObject;
import com.filenet.api.core.ObjectStore;
import com.filenet.api.core.ReferentialContainmentRelationship;
import com.filenet.api.core.UpdatingBatch;
import com.filenet.api.collection.FolderSet;
import com.filenet.api.collection.ReferentialContainmentRelationshipSet;
import com.filenet.api.exception.EngineRuntimeException;
import com.filenet.api.exception.ExceptionCode;
import com.filenet.api.property.FilterElement;
import com.filenet.api.property.Properties;
import com.filenet.api.property.Property;
import com.filenet.api.property.PropertyFilter;
import com.filenet.api.query.SearchSQL;
import com.filenet.api.query.SearchScope;
import com.filenet.api.security.Group;
import com.filenet.api.security.Permission;
import com.filenet.api.util.Id;
import com.filenet.api.core.VersionSeries;
import com.ibm.ce.rep.utils.CEReplicate;
import com.ibm.ce.utils.CEReplicateConfig;
import com.ibm.filenet.ce.CEConnection.CEConnection;
//import com.ibm.ce.utils.FolderUtil;
public class CEReplicate {
//ASB ... 2 010 Add static types for property type recognition
private static final int BinaryType = 1;
private static final int BoolType = 2;
private static final int DateType = 3;
private static final int DoubleType = 4;
private static final int GUIDType = 5;
private static final int LongType = 6;
private static final int ObjectType = 7;
private static final int StringType = 8;
//ASB ... 2 010 - End
//ASB ... 2 011 Add static types for property Cardinaity recognition
private static final int CardinalityENUM = 1;
private static final int CardinalityLIST = 2;
private static final int CardinalitySINGLE = 0;
//ASB ... 2 011 - End
private ObjectStore os;
private ObjectStore osinput;
private CEConnection con_MigrateDB;
private int maxMin; //ASB004 Moved Scope out to allow for correct storage of Document Version count
private CEConnection ceConn;
private UpdatingBatch ub;
private HashMap classNames;
private CEReplicateConfig cemc;
private static String ceContainmentNameExtOn; //ASB 01/03/2022
private SimpleDateFormat sdf = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss"); //ASB 16/02/2022 Update for 24 Hour clock
public long processedCount;
private long unfileProcessedCount;
private long fileProcessedCount; //ASB ... 4 030
private long updateProcessedCount;
private long updateSecurityProcessedCount; //ASB ... 4 021
private long updateProcessedDocCount;
private long retryTotalProcessedCount; //ASB ... 4 031
private static int retryLimit = 7; //ASB ... 4 031
//ASB 24/11/2021 - ADD initial count
private int batchCount = 0;
private Object group;
//ASB 24/11/2021 - ADD static
private static int batchSize = 250;
//ASB ... 2 021 Additional Replication Parameters
private String readOnlyGroup;
private Date startSearchDate;
private int maxSearchRecords;
//ASB ... 2 021 End
//ASB ... 4 022 Folder Symbolic Name List
private String folderSymbolList = "";
private String folderExcludeSymbolList = "";
private static Logger logger = Logger.getLogger(CEReplicate.class.getName());
/**
* @param args
*/
public static void main(String[] args) throws Exception {
logger.info("CE Replicate Version 3.7");
//ASB set up the Containment Name
CEReplicate ceReplicate = new CEReplicate();
ceReplicate.init();
long msecs = System.currentTimeMillis();
//ASB Comment Out Arguments - ALL in the same run now
/*
* if (args[0].compareToIgnoreCase("importfolders") == 0){
*/
logger.info("Begin Import Folders ...");
ceReplicate.importFolders();
msecs = System.currentTimeMillis() - msecs;
logger.info("Import " + ceReplicate.getProcessedCount() + " Folders took " + msecs + " milliseconds");
//}
msecs = System.currentTimeMillis();
//ASB ... 3 003 Add deletion processing - Documents
//ASB ... 4 011 Moved to delete documents before importing them!
/*
* if (args[0].compareToIgnoreCase("deletedocs") == 0){
*/
logger.info("Begin Delete Docs ...");
ceReplicate.deleteDocuments();
msecs = System.currentTimeMillis() - msecs;
logger.info("Delete " + ceReplicate.getProcessedCount() + " Docs took " + msecs + " milliseconds");
//}
/*
* if (args[0].compareToIgnoreCase("importdocs") == 0){
*/
msecs = System.currentTimeMillis();
logger.info("Begin Import Docs ...");
ceReplicate.importDocuments();
msecs = System.currentTimeMillis() - msecs;
logger.info("Import " + ceReplicate.getProcessedCount() + " Docs took " + msecs + " milliseconds");
//}
/*
* if (args[0].compareToIgnoreCase("deletefolders") == 0){
*/
//ASB ... 3 003 Add deletion processing - Folders
msecs = System.currentTimeMillis();
logger.info("Begin Delete Folders ...");
ceReplicate.deleteFolders();
msecs = System.currentTimeMillis() - msecs;
logger.info("Delete " + ceReplicate.getProcessedCount() + " Folders took " + msecs + " milliseconds");
//}
//ASB ... 4 012 New Update documents process
/*
* if (args[0].compareToIgnoreCase("updatedocs") == 0){
*/
msecs = System.currentTimeMillis();
logger.info("Begin Update Docs ...");
ceReplicate.updateDocuments();
msecs = System.currentTimeMillis() - msecs;
logger.info("Update " + ceReplicate.getUpdateProcessedDocCount() + " Docs took " + msecs + " milliseconds");
//}
//}
//}
}
private String getProcessedCount(){
return String.valueOf(processedCount);
}
private String getUnfileProcessedCount(){
return String.valueOf(unfileProcessedCount);
}
//ASB ... 4 030
private String getFileProcessedCount(){
return String.valueOf(fileProcessedCount);
}
private String getUpdateProcessedCount(){
return String.valueOf(updateProcessedCount);
}
private String getUpdateProcessedDocCount(){
return String.valueOf(updateProcessedDocCount);
}
//Deletes all the versions associated with the document
//Used when a document is being re-imported
private boolean deleteDoc(String docId) throws Exception
{
try
{
//ASB ... 4 019 Retrieve input doc to check the state
// Need to ignore checked out docs
Document docSource = Factory.Document.fetchInstance(osinput, docId, null);
if(docSource.get_VersionSeries().get_IsReserved()){
return false;
}
Document doc = null;
// S T A R T Deletion list
//Get Version Series ID
VersionableSet verSet = docSource.get_Versions();
int versionCount=0;
Iterator versIt = verSet.iterator();
int versCount = 0;
Id delId = null;
Document docDel = null; //Input Document versions
Id delDocFound = null;
VersionSeries vs = null; //ASB ... 4 23 16 11 2021
while (versIt.hasNext()){
versCount ++;
docDel = (Document)versIt.next();
//Get id to delete
delId = docDel.get_Id();
try{
doc = Factory.Document.fetchInstance(os, delId, null);
delDocFound = doc.get_Id();
vs = doc.get_VersionSeries();
break;
}catch(Exception delEx){
//Here because this version does not exist
}
}
// E N D
// Return in the following cases
// a) IsCurrentVersion = false
// b) Version series Current version is reserved
//ASB ... 4 23 VersionSeries vs = doc.get_VersionSeries();
//ASB ... 4 019 Check if the target doc is reserved and return if so!
// Most unlikely as it is supposed to be read-only!
//ASB ... 4 23if (vs.get_CurrentVersion().get_IsReserved()){
//ASB ... 4 23 return false;
//ASB ... 4 23 }
vs.delete();
vs.save(RefreshMode.REFRESH);
}
catch(Exception e)
{
logger.error("Error deleting document - " + docId);
logger.error(e.getMessage(), e);
throw e;
}
return true;
}
//Deletes all the versions associated with the document
//Used when a document is being re-imported
private boolean deleteDocTargetOnly(String docId) throws Exception
{
try
{
//ASB ... 4 019 Retrieve input doc to check the state
// Need to ignore checked out docs
Document doc = Factory.Document.fetchInstance(os, docId, null);
// S T A R T Deletion list
//Get Version Series ID
VersionableSet verSet = doc.get_Versions();
int versionCount=0;
Iterator versIt = verSet.iterator();
int versCount = 0;
Id delId = null;
Document docDel = null; //Input Document versions
Id delDocFound = null;
VersionSeries vs = null; //ASB ... 4 23 16 11 2021
while (versIt.hasNext()){
versCount ++;
docDel = (Document)versIt.next();
//Get id to delete
delId = docDel.get_Id();
try{
docDel = Factory.Document.fetchInstance(os, delId, null);
delDocFound = doc.get_Id();
vs = doc.get_VersionSeries();
break;
}catch(Exception delEx){
//Here because this version does not exist
}
}
// E N D
// Return in the following cases
// a) IsCurrentVersion = false
// b) Version series Current version is reserved
//ASB ... 4 23 VersionSeries vs = doc.get_VersionSeries();
//ASB ... 4 019 Check if the target doc is reserved and return if so!
// Most unlikely as it is supposed to be read-only!
//ASB ... 4 23if (vs.get_CurrentVersion().get_IsReserved()){
//ASB ... 4 23 return false;
//ASB ... 4 23 }
vs.delete();
vs.save(RefreshMode.REFRESH);
}
catch(Exception e)
{
logger.error("Error deleting document - " + docId);
logger.error(e.getMessage(), e);
throw e;
}
return true;
}
//ASB ... 3 003
//Deletes the Folder whose GUID is passed in
//Used when a folder has been deleted from the source Object Store
//And there is a deletion Event record for the Folder Class
private void deleteFolder(String folderId) throws Exception
{
try
{
Folder folder = Factory.Folder.fetchInstance(os, folderId, null);
folder.delete();
folder.save(RefreshMode.REFRESH);
}
catch(Exception e)
{
logger.error("Error deleting folder - " + folderId);
logger.error(e.getMessage(), e);
throw e;
}
}
//Main Process loop for the Import Documents process.
//Outer loop executes the retrieval of Documents from the source Object Store until no matches are returned.
//Inner loop processes each Document returned from the retrieval.
//
private void importDocuments() throws Exception {
BufferedWriter auditFileWriter;
Long importStartTime;
Long docImportStartTime;
Long timeToImport;
long maxRunTimeMillis;
//Long maxDocCount = new Long(0);
boolean moreDocsToImport=true;
Integer consecutiveErrs=0;
Integer errorCount=0;
boolean completed=false;
this.processedCount = 0;
importStartTime = System.currentTimeMillis();
maxRunTimeMillis = cemc.getImportMaxRunTimeMinutes()*60*1000;
Long maxDocCount = new Long(cemc.getImportMaxDocCount());
auditFileWriter = new BufferedWriter(new FileWriter(cemc.getImportAuditFile()));
auditFileWriter.write("Import Start - ");
auditFileWriter.newLine();
auditFileWriter.write("GUID, SOURCEVSID, DESTDOCID, STATE, Millisecs");
auditFileWriter.newLine();
while (moreDocsToImport && completed==false)
{
moreDocsToImport=false;
SearchSQL sqlObject = new SearchSQL();
//sqlObject.setSelectList("d.DocumentTitle, d.Id, d.Creator, d.DateCreated, d.LastModifier, d.DateLastModified, d.Name, d.Owner, d.IsReserved, d.IsCurrentVersion, d.IsFrozenVersion, d.IsVersioningEnabled, d.MajorVersionNumber, d.MinorVersionNumber,d.VersionStatus, d.IsInExceptionState, d.LockToken, d.LockTimeout, d.LockOwner, d.ReservationType,d.DateCheckedIn, d.StorageLocation, d.ContentElementsPresent, d.ContentSize, d.MimeType, d.DateContentLastAccessed, d.ContentRetentionDate, d.CurrentState, d.ClassificationStatus, d.IndexationId, d.CmIndexingFailureCode, d.CompoundDocumentState, d.CmRetentionDate, d.ComponentBindingLabel,d.EntryTemplateObjectStoreName, d.EntryTemplateLaunchedWorkflowNumber, d.EntryTemplateId, d.PublicationInfo, d.PublishingSubsidiaryFolder, d.IgnoreRedirect, d.FoldersFiledIn");
//ASB ... 4 006 Need to allow for additional properties required
sqlObject.setSelectList("d.*");
// sqlObject.setSelectList("d.DocumentTitle, d.Id, d.Creator, d.DateCreated, d.LastModifier, d.DateLastModified, d.Name, d.Owner, d.IsReserved, d.IsCurrentVersion, d.IsFrozenVersion, d.IsVersioningEnabled, d.MajorVersionNumber, d.MinorVersionNumber,d.VersionStatus, d.IsInExceptionState, d.LockToken, d.LockTimeout, d.LockOwner, d.ReservationType,d.DateCheckedIn, d.StorageLocation, d.ContentElementsPresent, d.ContentSize, d.MimeType, d.DateContentLastAccessed, d.ContentRetentionDate, d.CurrentState, d.ClassificationStatus, d.IndexationId, d.CmIndexingFailureCode, d.CompoundDocumentState, d.CmRetentionDate, d.ComponentBindingLabel,d.EntryTemplateObjectStoreName, d.EntryTemplateLaunchedWorkflowNumber, d.EntryTemplateId, d.PublicationInfo, d.PublishingSubsidiaryFolder, d.IgnoreRedirect, d.FoldersFiledIn");
//ASB ... 2 022
Integer maxRecords = cemc.getExportMaxDocCount();
sqlObject.setMaxRecords(maxRecords); //was hard coded to 20 !
sqlObject.setFromClauseInitialValue("Document", "d", true); //set true to include subclasses
//ASB ... 022 - Retrieve Date start from config.xml
String sDeltaHours = cemc.getDeltaHours();
String sSearchDate = cemc.getStartSearchDate();
//ASB ... 4 002 Print Out the Current Date used for searching
//System.out.print("
Start Search Date : " + sSearchDate);
logger.info("Start Search Date : " + sSearchDate);
logger.info("Start Delta Hours : " + sDeltaHours);
//ASB ... 4 002 Now dynamically update the next Replication Date
cemc.updateProcessDate(new Date());
//Date searchDate = sdf.parse(sSearchDate);
//sqlObject.setWhereClause("d.DateCreated > " + "20210923T125628Z and d.IsCurrentVersion = true" );
//ASB ... 022 - enter date to search
//ASB ... 4 019 Exclude reservation objects
//sqlObject.setWhereClause("d.DateCreated > " + sSearchDate.trim() + " and d.IsCurrentVersion = true and d.IsReserved = false" ); //ASB ... 4 003
//ASB ... 4 023 Set documents to be imported based on Date Last Modified
// Rather than DateCreated to ensure documents left checked out passed the delta time are included when checked in !!
sqlObject.setWhereClause("d.DateLastModified > " + sSearchDate.trim() + " and d.IsCurrentVersion = true and d.IsReserved = false" ); //ASB ... 4 003
sqlObject.setOrderByClause("d.DateLastModified,d.MajorVersionNumber, d.MinorVersionNumber");
// Check the SQL statement.
logger.info("SQL: " + sqlObject.toString());
// Create a SearchScope instance. (Assumes you have the object store
// object.)
SearchScope search = new SearchScope(osinput);
// Set the page size (Long) to use for a page of query result data. This value is passed
// in the pageSize parameter. If null, this defaults to the value of
// ServerCacheConfiguration.QueryPageDefaultSize.
Integer myPageSize = new Integer(1000);
// Specify a property filter to use for the filter parameter, if needed.
// This can be null if you are not filtering properties.
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 1;
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_LIST, null));
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
// Set the (Boolean) value for the continuable parameter. This indicates
// whether to iterate requests for subsequent pages of result data when the end of the
// first page of results is reached. If null or false, only a single page of results is
// returned.
Boolean continuable = new Boolean(true);
// Execute the fetchObjects method using the specified parameters.
IndependentObjectSet myObjects = search.fetchObjects(sqlObject, myPageSize, myFilter, continuable);
// You can then iterate through the collection of rows to access the properties.
int rowCount = 0;
Iterator iter = myObjects.iterator();
Id GUID = null;
//ASB ... 4 019 Need to set completed to true in the createversions method
while (iter.hasNext() && completed==false){
//moreDocsToImport=true; Assume All brought in in one batch At the moment ASB ...2 .001
docImportStartTime = System.currentTimeMillis();
Boolean importFailure = false;
Boolean filedDoc = false;
Document d = null;
//String fileName = rs.getString("EXPORTFILE").trim();
String documentTitle = "";
//ASB ... 3 005 - Retrieve Folder list for checks
// - (Only replicate documents in config Path)
String [] sDocFolderPath = new String [1000]; //TODO set max document Folders Filed In from config.xml
int folderCount = -1;
//String guid = rs.getString("GUID").trim(); //ASB 11/03/2021 - Added Trimmed string
//ASB ... 1 Add to pass to new version of the createDocument Method
IndependentObject object = (IndependentObject) iter.next();
//ASB Software Development Limited example found to illustrate full property retrieval 01-08-2022
// Id id = document.get_Id(); //get the doc ID back on this shallow document copy.
// *retrieve a complete document copy back with complete property set using doc ID*/
// Document docCopy = (Document)Factory.Document.fetchInstance(objectStore, id, null);
// FolderSet pFolders = doc.get_FoldersFiledIn();//Get the full "FolderFiledIn" property on the new copy
//
// ASB Properties props = object.getProperties();
//ASB Iterator iterProps = props.iterator();
GUID = ((Document) object).get_Id();
//ASB Software Development Limited - get full document property set which we need! 01-08-2022
Document docCopy = (Document)Factory.Document.fetchInstance(osinput, GUID, null);
String guid = GUID.toString().trim();
Boolean isCurrent = ((Document) docCopy).get_IsCurrentVersion();
Boolean isReserved = ((Document) docCopy).get_IsReserved();
Properties props = docCopy.getProperties();
Iterator iterProps = props.iterator();
//Fetch object
String lastGoodPropName = "";
String propName = "";
while (iterProps.hasNext() )
{
// ASB ... 4 0018 Additional try/catch block to
// Trap property error here!
try{
com.filenet.api.property.Property prop = (com.filenet.api.property.Property)iterProps.next();
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("Property: " + prop.getPropertyName() );
}
propName = prop.getPropertyName(); //ASB ... 4 018 For error
if ( ((com.filenet.api.property.Property) prop).getObjectValue() != null )
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info(" Value: " + prop.getObjectValue().toString() );
}
if (prop.getPropertyName().equalsIgnoreCase("DocumentTitle"))
{
documentTitle = prop.getStringValue();
}
//GET FOLDERS FILED IN FOR THIS DOC
if (prop.getPropertyName().equalsIgnoreCase("FoldersFiledIn"))
{
if ( prop.getObjectValue() != null )
{
FolderSet fs = (FolderSet)prop.getIndependentObjectSetValue();
Iterator iterFs = fs.iterator();
//ASB ... 3 005 - Retrieve Folder list for checks
// - (Only replicate documents in config Path)
while (iterFs.hasNext())
{
folderCount ++;
Folder folder = (Folder)iterFs.next();
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("Folder Name: " + folder.get_FolderName() +
" Folder Path: " + folder.get_PathName());
}
sDocFolderPath[folderCount] = folder.get_PathName();
filedDoc = true;
}
}
}
}catch(Exception errprop){
logger.info("Document GUID :" + guid + " : Property " + propName + " Caused Error : " + errprop.getMessage() + " : Last Good property :" + lastGoodPropName);
}
lastGoodPropName = propName;
}
//E N D -- DEBUG OFF/ON SECTION
//ASB ... 3 005 - Retrieve Folder list for checks
// - (Only replicate documents in config Path)
int checkFolderCount = -1;
Boolean filterDoc = false;
while(checkFolderCount < folderCount){
checkFolderCount ++;
if ((sDocFolderPath[checkFolderCount] != null) && !(sDocFolderPath[checkFolderCount].startsWith( cemc.getImportOSRootFoler()))){
//Document selected is new but not in our import Folder Path!
logger.info("IGNORED: New Document: " + documentTitle + " found linked to external Folder Path : " + sDocFolderPath[checkFolderCount] + " Should be linked to :" + cemc.getImportOSRootFoler());
filterDoc = true;
break;
}
}
try {
//isCurrent=false; //ASB ... 1
if(!filedDoc){
//Document selected is new but not in our import Folder Path!
logger.info("IGNORED: New Document: " + documentTitle + " not linked to any Folder Path ! ");
this.processedCount = this.processedCount + 1;
}
if (!importFailure && !filterDoc && filedDoc)
{
// Add Version series ID and ID to retain the GUIDs
// NB already set-up in GUID and VSID
docImportStartTime = System.currentTimeMillis();
//ASB Use docCopy now
//ASB d = createDocument(documentTitle, GUID, object, props, iterProps, isCurrent, isReserved,auditFileWriter); // ASB 01-08-2022
d = createDocument(documentTitle, GUID, docCopy, props, iterProps, isCurrent, isReserved,auditFileWriter);
//ASB ... 4 019 d is null at this point if all the versions have been created
Document doc = Factory.Document.fetchInstance(os, GUID, null);
Long runTimeMillis = System.currentTimeMillis() - importStartTime;
if (this.processedCount == maxDocCount && maxDocCount>0)
{
completed = true;
logger.info("Processing Completed Max Document Count Reached. ");
}
else if ((runTimeMillis > maxRunTimeMillis)&& maxRunTimeMillis>0 )
{
completed = true;
logger.info("Processing Completed Max Processing Time Reached.");
}
consecutiveErrs=0; // success so initialise the consecutive errs count
}
}
catch (Exception ceme){
//ASB ... 4 Need to check if d is null, if so then not created!
try { // getting occasional unexpected errors on the server, try to make sure doc is deleted
if(d != null){ // ASB ... 4 008 Then valid to delete d otherwise it did not get creaetd
String docId = d.get_Id().toString();
deleteDoc(docId);
//ASB ... 2 013 - Set Import failure to true here
importFailure = true;
consecutiveErrs ++;
errorCount++;
logger.warn(ceme.getMessage());
if (consecutiveErrs > cemc.getImportMaxErrorCount())
{
auditFileWriter.write("Maximum number of consecutive errors reached - Exiting.");
auditFileWriter.newLine();
logger.info("Maximum number of consecutive errors reached - Exiting.");
throw ceme;
}
}
else{
//Assume we need to delete the Current Document GUID
//String docId = GUID.toString();
//deleteDoc(docId);
}
}catch (Exception e){
// }
//ASB ... 4 019 If d is null There are no errors ... moved message inside the try block above
}
}
}
auditFileWriter.write("Finished - time taken " + (System.currentTimeMillis() - importStartTime) + "milliseconds");
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents processed - " + String.valueOf(processedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents failed to be processed - " + errorCount.toString());
auditFileWriter.newLine();
//ASB ... 4 031 Add retry count for Content
auditFileWriter.write("Finished - document total 'Add Content Retry Count' - " + String.valueOf(retryTotalProcessedCount));
auditFileWriter.newLine();
auditFileWriter.close();
}
}
//Creates a org.w3c.dom.Document object
//Loops through each version calling createVersion to create the document /add the version
//to it.
private Document createDocument(String documentTitle, Id GUID, IndependentObject object, Properties props, Iterator iterProps,Boolean isCurrent, Boolean isReserved, BufferedWriter auditFileWriter) throws Exception,EngineRuntimeException {
try
{
Document currentDocument = (Document) object;
String documentclass = currentDocument.getClassName();
FolderSet nFolders = currentDocument.get_FoldersFiledIn();
String sCreator = currentDocument.get_Creator();
Date dateCreated = currentDocument.get_DateCreated();
Date dateLastModified = currentDocument.get_DateLastModified();
Date dateCheckedIn = currentDocument.get_DateCheckedIn();
String sModifier = currentDocument.get_LastModifier();
String sOwner = currentDocument.get_Owner();
Boolean bCurrent = currentDocument.isCurrent();
Long docImportStartTime;
Long timeToImport;
docImportStartTime = System.currentTimeMillis();
//ASB ... 2 014 Get the Version Numbers of the Document
Integer docMajorVersion = currentDocument.get_MajorVersionNumber();
Integer docMinorVersion = currentDocument.get_MinorVersionNumber();
//ASB ... 2 015 Check if this document has previous versions
isCurrent = bCurrent; //ASB ... 1
//ASB ... 2 007 Fetch the Source Object Document to be re-created
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 2; //Changed from 4 to 2 -- TODO Set as a parameter
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Document dInput = Factory.Document.fetchInstance(osinput, GUID, myFilter);
VersionSeries versions =dInput.get_VersionSeries();
//ASB ... 2 Get the version Series ID and number of Document Versions and set it in the copy
Id vGUID = versions.get_Id();
//this.minMax = 0; // Set to return maximum value of the minimum version
Integer docVersionCount = countVersions(versions);
//ASB ... 2 016 Need to create an array of document versions from oldest to
// newest since the iterator returns the newest document first!
Document[][] docVersions = new Document[docVersionCount +1][maxMin+1];
docVersions = getDocumentVersionsArray (versions,docVersions, maxMin);
Document d = null;
if (!docVersionCount.equals(1)){
//Create versions here
createVersions(documentTitle, GUID, object, props, iterProps, isCurrent, isReserved, docVersions,docVersionCount,maxMin,auditFileWriter);
} else {
//ASB ... 3 006 New Version of the first document create to store Version series ID
//ASB ... 3 006 Many thanks to David Greenhouse for supplying the call :-)
//ASB ... 4 008
try {
docImportStartTime = System.currentTimeMillis();
d = Factory.Document.createInstance(os, documentclass, GUID, vGUID, com.filenet.api.constants.ReservationType.EXCLUSIVE);
}catch(Exception e)
{
//If the Exception is this exists then delete and retry add again
logger.error("Document already exists: " + documentTitle + "");
boolean deletedDoc = false;
//docImportStartTime = System.currentTimeMillis();
deletedDoc = this.deleteDoc(GUID.toString());
//ASB ... 4 008 And retry
d = Factory.Document.createInstance(os, documentclass, GUID, vGUID, com.filenet.api.constants.ReservationType.EXCLUSIVE);
//ASB ... 4 023
//String docId = GUID.toString();
//String sourceVSID = vGUID.toString();
//Write to audit log
//timeToImport = System.currentTimeMillis() - docImportStartTime;
//auditFileWriter.write(docId + "," + sourceVSID + "," + docId + ",Imported," + timeToImport.toString());
//auditFileWriter.newLine();
//this.processedCount = this.processedCount +1;
}
//Create Document with the same GUID as the Source Object
//ASB ... 2 002 : Phase 2 - Add properties from Iterator
writeDocProps(d, props, currentDocument);
d.set_DateCreated(dateCreated); //ASB 17/02/2022
d.set_Owner(sOwner);
d.save(RefreshMode.REFRESH); //ASB XXX
this.processedCount = this.processedCount +1;
String guid = GUID.toString().trim();
//Write to audit log
timeToImport = System.currentTimeMillis() - docImportStartTime;
auditFileWriter.write(guid + "," + vGUID.toString() + "," + d.get_Id().toString() + ",Imported," + timeToImport.toString());
auditFileWriter.newLine();
//ADD CONTENT
//ASB ... 4 024 -JUST ADDED
//ASB ... 4 031 Add retry 7 times to fix API bug!!
int retryCount = 0;
Document currDoc = null;
while (retryCount < retryLimit){
try {
currDoc = (Document)osinput.fetchObject(documentclass, GUID, myFilter);
//Document currDoc = (Document)osinput.fetchObject("Document", GUID, myFilter);
ContentTransfer ct = Factory.ContentTransfer.createInstance();
InputStream str = currDoc.accessContentStream(0);
ct.setCaptureSource(str);
// Add Document Title
ct.set_RetrievalName(documentTitle);
// Add Content Mime Type
ct.set_ContentType(currDoc.get_MimeType());
ContentElementList cel = Factory.ContentElement.createList();
cel.add(ct);
d.set_ContentElements(cel);
break;
}
catch (Exception e){
String testMessage = e.getMessage();
String testCode = e.toString();
if( testMessage.contains("A uniqueness requirement has been violated")){
retryCount = retryLimit;
break;
}
//ASB ... 4 Check if the Content could not be added because the document already exists, if so
// Remove document and retry (next run)
// ASB ... 4 025 Log Exception here
logger.error(e.getMessage(), e);
retryCount ++;
retryTotalProcessedCount ++;
logger.info("Retry Count - createDocument method : " + retryCount + " of " + retryLimit + " On : " + documentTitle + " : with GUID : " + GUID);
}
} //ASB ... 4 031 Above loops Test retryCount less than the retryLimit of 7
//Add security from the Source Document
//ASB ... 2 020 Update Document ACL from source Document security
AccessPermissionList aclIn = currDoc.get_Permissions();
//ASB ... 2 021 Add in Read Only Group
// Add the permission to the list for the Object Store.
//Get The group and user from the config.xml file
//
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
d.set_Permissions(aclIn);
Boolean isMajorVersion = false;
if(currDoc.get_MinorVersionNumber() == 0) isMajorVersion = true;
//ASB ... 4 028 Change to do not Classify from AutoClassify.AUTO_CLASSIFY, to
if (isMajorVersion){
d.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MAJOR_VERSION);
}
else {
d.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MINOR_VERSION);
}
//d.set_DateCreated(dateCreated); //ASB ... 4 029
d.set_DateLastModified(dateLastModified);
d.set_LastModifier(sModifier);
d.set_DateCheckedIn(dateCheckedIn); //ASB ... 2 005
d.save(RefreshMode.REFRESH); //ASB XXX
FolderSet fs = currentDocument.get_FoldersFiledIn();
Iterator iterFs = fs.iterator();
while (iterFs.hasNext())
{
Folder folder = (Folder)iterFs.next();
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("
Folder Name: " + folder.get_FolderName() +
" Folder Path: " + folder.get_PathName());
}
folderDoc(d,folder.get_PathName(),documentTitle);
}
} //ASB ... 2 014 Count of Versions of Documents
return d;
}catch(EngineRuntimeException ere)
{
//ASB ... 2 005
// Create failed. See if it's because the Document exists.
ExceptionCode code = ere.getExceptionCode();
if (code.getErrorId() != ExceptionCode.DB_NOT_UNIQUE.getErrorId() )
{
logger.error("Unexpected Error : " + documentTitle + " Error stack: " + ere.getStackTrace());
throw ere;
}
logger.error("Document already exists: " + documentTitle + "");
//ASB ... 2 006 Delete document in the Target Object Store and Recreate
this.deleteDoc(GUID.toString());
//ASB ... 2 007 Fetch the Source Object Document to be re-created
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 2; //Changed from 4 to 2 -- TODO Set as a parameter
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Document currentDocument = (Document) object;
String documentclass = currentDocument.getClassName();
Document dInput = Factory.Document.fetchInstance(osinput, GUID, myFilter);
//Create Version Series for the document in the Target Object Store
VersionSeries versions = dInput.get_VersionSeries();
//Get the Document Versions to be added from the Source Document
VersionableSet verSet = versions.get_Versions();
Iterator versIt = verSet.iterator();
int versCount = 0;
int majorVn = 0;
int minorVn = 0;
Document doc = null; //Input Document versions
Document d = null; //Output document versions
isReserved = false;
Id createdGuid = null;
while (versIt.hasNext()){
versCount ++;
doc = (Document)versIt.next();
majorVn = doc.get_MajorVersionNumber();
minorVn = doc.get_MinorVersionNumber();
if (!isReserved)isReserved = doc.get_IsReserved();
Id nextGUID = doc.get_Id();
//Check for error here (sometimes!)
d = createVersion(versCount, d, doc, isReserved, majorVn,minorVn, documentTitle, GUID, createdGuid, nextGUID,auditFileWriter);
createdGuid = d.get_Id();
if (versCount == 1){
FolderSet fs = doc.get_FoldersFiledIn();
Iterator iterFs = fs.iterator();
while (iterFs.hasNext())
{
Folder folder = (Folder)iterFs.next();
//Create/Check and Make link in Target for the Document to this Folder
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("
Folder Name: " + folder.get_FolderName() +
" Folder Path: " + folder.get_PathName());
}
folderDoc(d,folder.get_PathName(),documentTitle);
}
}
if (isReserved){
break;
}
}
if (isReserved)
d = doReservationProperties(versCount, doc, d, isReserved, majorVn,minorVn,documentTitle);
//ASB ... 2 020 Update Document ACL from source Document security
AccessPermissionList aclIn = dInput.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//ASB ... 4 027 aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
d.set_Permissions(aclIn);
return d;
}
}
//This method returns the count of the number of Document versions
//(Unfortunately java Iterator class does not return size!)
private int countVersions(VersionSeries versions){
int versionCount=0;
VersionableSet verSet = versions.get_Versions();
Iterator versIt = verSet.iterator();
int versCount = 0;
maxMin = 0;
int min = 0;
Document doc = null; //Input Document versions
while (versIt.hasNext()){
versCount ++;
doc = (Document)versIt.next();
//Record largest minor version value for loop
min = doc.get_MinorVersionNumber();
if (min > maxMin){
this.maxMin = min;
}
}
versionCount = versCount;
return versionCount;
}
//Main Process loop for the Delete Documents process.
//Outer loop executes the retrieval of Documents from the source Object Store Events until no matches are returned.
//Inner loop processes each Document Deletion Event returned from the retrieval.
//ASB ... 3 003 deleteDocuments Method
private void deleteDocuments() throws Exception {
Id GUID = null;
BufferedWriter auditFileWriter;
Long deleteStartTime;
Long docDeleteStartTime;
Long timeToDelete;
long maxRunTimeMillis;
boolean moreDocsToDelete=true;
Integer consecutiveErrs=0;
Integer errorCount=0;
boolean completed=false;
processedCount = 0;
deleteStartTime = System.currentTimeMillis();
maxRunTimeMillis = cemc.getImportMaxRunTimeMinutes()*60*1000;
Long maxDocCount = new Long(cemc.getDeleteMaxDocCount());
auditFileWriter = new BufferedWriter(new FileWriter(cemc.getDeleteDocumentAuditFile()));
auditFileWriter.write("Delete Start - ");
auditFileWriter.newLine();
auditFileWriter.write("GUID, INITIATING USER, EVENT TYPE, AUDIT SEQUENCE");
auditFileWriter.newLine();
while (moreDocsToDelete && completed==false)
{
/*
Query to use is as follows:
SELECT [EVENT].[This], [EVENT].[CmAuditSequence], [EVENT].[ClassDescription], [EVENT].[Creator], [EVENT].[DateCreated], [EVENT].[SourceObjectId],
[EVENT].[DateLastModified], [EVENT].[EventStatus], [EVENT].[Id], [EVENT].[InitiatingUser], [EVENT].[LastModifier],
[EVENT].[Name], [EVENT].[Owner]
FROM [EVENT] INNER JOIN [CLASSDEFINITION]
ON [EVENT].SourceClassId = [CLASSDEFINITION].[Id]
WHERE ([EVENT].[DateCreated] >= 20210920T210827Z) AND ([ClassDefinition].[SymbolicName] <>'Folder')
ORDER BY [EVENT].[DateCreated]
*/
moreDocsToDelete=false;
SearchSQL sqlObject = new SearchSQL();
sqlObject.setSelectList("e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId");
//ASB ... 2 022
Integer maxRecords = cemc.getDeleteMaxDocCount();
sqlObject.setMaxRecords(maxRecords); //was hard coded to 20 !
sqlObject.setFromClauseInitialValue("Event", "e", true);
sqlObject.setFromClauseAdditionalJoin(JoinOperator.INNER, "CLASSDEFINITION", "cd", "e.SourceClassId", JoinComparison.EQUAL, "cd.Id", true);
//ASB ... 022 - Retrieve Date start from config.xml
String sSearchDate = cemc.getStartSearchDate();
//ASB ... 022 - enter date to search
//ASB ... 4 013 Changed to cd.SymbolicName <> 'Folder'
// sqlObject.setWhereClause("(e.DateCreated > " + sSearchDate.trim() + ") and (cd.SymbolicName = 'Document')" );
//ASB ... 4 017 Changed to filter for just Success Audit Events
sqlObject.setWhereClause("(e.DateCreated >= " + sSearchDate.trim() + ")" + folderExcludeSymbolList + " AND (e.EventStatus = 0)" );
sqlObject.setOrderByClause("e.CmAuditSequence,e.DateCreated");
// Check the SQL statement.
logger.info("SQL: " + sqlObject.toString());
// Create a SearchScope instance. (Assumes you have the object store
// object.)
SearchScope search = new SearchScope(osinput);
// Set the page size (Long) to use for a page of query result data. This value is passed
// in the pageSize parameter. If null, this defaults to the value of
// ServerCacheConfiguration.QueryPageDefaultSize.
Integer myPageSize = new Integer(1000);
// Specify a property filter to use for the filter parameter, if needed.
// This can be null if you are not filtering properties.
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 1;
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_LIST, null));
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
// Set the (Boolean) value for the continuable parameter. This indicates
// whether to iterate requests for subsequent pages of result data when the end of the
// first page of results is reached. If null or false, only a single page of results is
// returned.
Boolean continuable = new Boolean(true);
// Execute the fetchObjects method using the specified parameters.
IndependentObjectSet myObjects = search.fetchObjects(sqlObject, myPageSize, myFilter, continuable);
// You can then iterate through the collection of rows to access the properties.
int rowCount = 0;
Iterator iter = myObjects.iterator();
while (iter.hasNext() && completed==false){
//moreDocsToImport=true; Assume All brought in in one batch At the moment ASB ...2 .001
docDeleteStartTime = System.currentTimeMillis();
Boolean deleteFailure = false;
//Document d = null;
String documentTitle = "";
//String guid = rs.getString("GUID").trim(); //ASB 11/03/2022 - Added Trimmed string
//ASB ... 1 Add to pass to new version of the createDocument Method
IndependentObject object = (IndependentObject) iter.next();
//ClassDescription object.get_ClassDescription();
Properties props = object.getProperties(); //TODO ASB ... 3
String eventDesc = "unknown";
Iterator prop = props.iterator();
String initiatingUser = "";
String eventSequence = "";
while (prop.hasNext()){
Property eventProp = (Property) prop.next();
//ASB ... 3 004 Get Event Type
if (eventProp.getPropertyName().equalsIgnoreCase("ClassDescription")){
eventDesc = eventProp.getEngineObjectValue().getProperties().get("SymbolicName").getStringValue();
}
//ASB ... 3 003 Check for the Document ID property to delete
if (eventProp.getPropertyName().equalsIgnoreCase("SourceObjectId")){
GUID = eventProp.getIdValue();
}
if (eventProp.getPropertyName().equalsIgnoreCase("InitiatingUser")){
initiatingUser = eventProp.getStringValue();
}
if (eventProp.getPropertyName().equalsIgnoreCase("CmAuditSequence")){
eventSequence = eventProp.getFloat64Value().toString();
}
} //prop.hasNet()
String guid = GUID.toString().trim();
boolean docDeleted = false;
try {
if (eventDesc.equalsIgnoreCase("CheckinEvent") ){
//ASB ... 3 003 Delete the Target document with the Delete event/Checkin Event in the Source
docDeleted = deleteDoc(GUID.toString()); //ASB ... 4 019 - Test if reservation Object
if (docDeleted){
auditFileWriter.write(GUID.toString() + ", " + initiatingUser+ ", " + eventDesc + ", " + eventSequence);
auditFileWriter.newLine();
processedCount++;
}
}
//ASB ... 4 023 Split to deal with DeletionEvent separately
if (eventDesc.equalsIgnoreCase("DeletionEvent") ){
//ASB ... 3 003 Delete the Target document with the Delete event/Checkin Event in the Source
docDeleted = deleteDocTargetOnly(GUID.toString()); //ASB ... 4 019 - Test if reservation Object
if (docDeleted){
auditFileWriter.write(GUID.toString() + ", " + initiatingUser+ ", " + eventDesc + ", " + eventSequence);
auditFileWriter.newLine();
processedCount++;
}
}
}
//catch (CEMigrateException ceme){
catch (Exception ceme){
try { // getting occasional unexpected errors on the server, try to make sure doc is deleted
String docId = GUID.toString();
//deleteDoc(docId);
//ASB ... 2 013 - Set Delete failure to true here
deleteFailure = true;
}catch (Exception e){
//
}
consecutiveErrs ++;
errorCount++;
logger.warn(ceme.getMessage());
if (consecutiveErrs > cemc.getDeleteMaxErrorCount())
{
auditFileWriter.write("Maximum number of consecutive delete errors reached - Exiting.");
auditFileWriter.newLine();
logger.info("Maximum number of consecutive errors reached - Exiting.");
throw ceme;
}
}
}
auditFileWriter.write("Finished - time taken " + (System.currentTimeMillis() - deleteStartTime) + "milliseconds");
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents deleted - " + String.valueOf(processedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents updated - " + String.valueOf(updateProcessedDocCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents failed to be deleted - " + errorCount.toString());
auditFileWriter.newLine();
auditFileWriter.close();
}
}
//Main Process loop for the Update Documents process.
//Outer loop executes the retrieval of Documents from the source Object Store Events until no matches are returned.
//Inner loop processes each Document Update Event returned from the retrieval.
//ASB ... 4 012 updateDocuments Method
private void updateDocuments() throws Exception {
Id GUID = null;
BufferedWriter auditFileWriter;
Long updateStartTime;
Long docUpdateStartTime;
Long timeToUpdate;
long maxRunTimeMillis;
boolean moreDocsToUpdate=true;
Integer consecutiveErrs=0;
Integer errorCount=0;
boolean completed=false;
processedCount = 0;
updateStartTime = System.currentTimeMillis();
maxRunTimeMillis = cemc.getImportMaxRunTimeMinutes()*60*1000;
Long maxDocCount = new Long(cemc.getUpdateMaxDocCount());
auditFileWriter = new BufferedWriter(new FileWriter(cemc.getUpdateDocumentAuditFile()));
auditFileWriter.write("Update Start - ");
auditFileWriter.newLine();
auditFileWriter.write("GUID, INITIATING USER, EVENT TYPE, AUDIT SEQUENCE");
auditFileWriter.newLine();
while (moreDocsToUpdate && completed==false)
{
/*
Query to use is as follows:
SELECT [EVENT].[This], [EVENT].[CmAuditSequence], [EVENT].[ClassDescription], [EVENT].[Creator], [EVENT].[DateCreated], [EVENT].[SourceObjectId],
[EVENT].[DateLastModified], [EVENT].[EventStatus], [EVENT].[Id], [EVENT].[InitiatingUser], [EVENT].[LastModifier],
[EVENT].[Name], [EVENT].[Owner]
FROM [EVENT] INNER JOIN [CLASSDEFINITION]
ON [EVENT].SourceClassId = [CLASSDEFINITION].[Id]
WHERE ([EVENT].[DateCreated] >= 20210920T210827Z) AND [ClassDefinition].[SymbolicName]='Document'
ORDER BY [EVENT].[DateCreated
*/
moreDocsToUpdate=false;
SearchSQL sqlObject = new SearchSQL();
sqlObject.setSelectList("e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId");
//ASB ... 2 022
Integer maxRecords = cemc.getUpdateMaxDocCount();
sqlObject.setMaxRecords(maxRecords); //was hard coded to 20 !
sqlObject.setFromClauseInitialValue("Event", "e", true);
sqlObject.setFromClauseAdditionalJoin(JoinOperator.INNER, "CLASSDEFINITION", "cd", "e.SourceClassId", JoinComparison.EQUAL, "cd.Id", true);
//ASB ... 022 - Retrieve Date start from config.xml
String sSearchDate = cemc.getStartSearchDate();
//ASB ... 022 - enter date to search
//sqlObject.setWhereClause("(e.DateCreated > " + sSearchDate.trim() + ") and (cd.SymbolicName = 'Document')" );
//ASB ... 4 017 Search limit to Success Audit Events
//ASB ... 4 022 Changed " and (cd.SymbolicName <> 'Folder')" to allow multiple sub-classes of Folder
sqlObject.setWhereClause("(e.DateCreated > " + sSearchDate.trim() + ")" + folderExcludeSymbolList + " AND (e.EventStatus = 0)" );
sqlObject.setOrderByClause("e.CmAuditSequence,e.DateCreated");
// Check the SQL statement.
logger.info("SQL: " + sqlObject.toString());
// Create a SearchScope instance. (Assumes you have the object store
// object.)
SearchScope search = new SearchScope(osinput);
// Set the page size (Long) to use for a page of query result data. This value is passed
// in the pageSize parameter. If null, this defaults to the value of
// ServerCacheConfiguration.QueryPageDefaultSize.
Integer myPageSize = new Integer(1000);
// Specify a property filter to use for the filter parameter, if needed.
// This can be null if you are not filtering properties.
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 1;
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
// Set the (Boolean) value for the continuable parameter. This indicates
// whether to iterate requests for subsequent pages of result data when the end of the
// first page of results is reached. If null or false, only a single page of results is
// returned.
Boolean continuable = new Boolean(true);
// Execute the fetchObjects method using the specified parameters.
IndependentObjectSet myObjects = search.fetchObjects(sqlObject, myPageSize, myFilter, continuable);
// You can then iterate through the collection of rows to access the properties.
int rowCount = 0;
Iterator iter = myObjects.iterator();
while (iter.hasNext() && completed==false){
//moreDocsToImport=true; Assume All brought in in one batch At the moment ASB ...2 .001
docUpdateStartTime = System.currentTimeMillis();
Boolean updateFailure = false;
//Document d = null;
String documentTitle = "";
//String guid = rs.getString("GUID").trim(); //ASB 11/03/2022 - Added Trimmed string
//ASB ... 1 Add to pass to new version of the createDocument Method
IndependentObject object = (IndependentObject) iter.next();
//ClassDescription object.get_ClassDescription();
Properties props = object.getProperties(); //TODO ASB ... 3
String eventDesc = "unknown";
Iterator prop = props.iterator();
String initiatingUser = "";
String eventSequence = "";
while (prop.hasNext()){
Property eventProp = (Property) prop.next();
//ASB ... 3 004 Get Event Type
if (eventProp.getPropertyName().equalsIgnoreCase("ClassDescription")){
eventDesc = eventProp.getEngineObjectValue().getProperties().get("SymbolicName").getStringValue();
}
//ASB ... 3 003 Check for the Document ID property to update
if (eventProp.getPropertyName().equalsIgnoreCase("SourceObjectId")){
GUID = eventProp.getIdValue();
}
if (eventProp.getPropertyName().equalsIgnoreCase("InitiatingUser")){
initiatingUser = eventProp.getStringValue();
}
if (eventProp.getPropertyName().equalsIgnoreCase("CmAuditSequence")){
eventSequence = eventProp.getFloat64Value().toString();
}
} //prop.hasNet()
String guid = GUID.toString().trim();
try {
//ASB ... 4 011 Check The Event Type is a UpdateEvent for the Document
//ASB ... 4 021 Update Security on Document Objects
if (eventDesc.equalsIgnoreCase("UpdateEvent")||eventDesc.equalsIgnoreCase("UpdateSecurityEvent")){
//Update of the Folder Detected - copy the new Source Folder properties
updateDocument(GUID);
auditFileWriter.write(GUID.toString() + ", " + initiatingUser+ ", " + eventDesc + ", " + eventSequence);
auditFileWriter.newLine();
updateProcessedDocCount++;
}
}
//catch (CEMigrateException ceme){
catch (Exception ceme){
try { // getting occasional unexpected errors on the server, try to make sure doc is updated
String docId = GUID.toString();
//ASB ... 2 013 - Set update failure to true here
updateFailure = true;
}catch (Exception e){
//swallow, don't care
}
consecutiveErrs ++;
errorCount++;
logger.warn(ceme.getMessage());
if (consecutiveErrs > cemc.getUpdateMaxErrorCount())
{
auditFileWriter.write("Maximum number of consecutive update errors reached - Exiting.");
auditFileWriter.newLine();
logger.info("Maximum number of consecutive errors reached - Exiting.");
throw ceme;
}
}
}
auditFileWriter.write("Finished - time taken " + (System.currentTimeMillis() - updateStartTime) + "milliseconds");
auditFileWriter.newLine();
//auditFileWriter.write("Finished - documents updated - " + String.valueOf(processedCount));
//auditFileWriter.newLine();
auditFileWriter.write("Finished - documents updated - " + String.valueOf(updateProcessedDocCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - documents failed to be updated - " + errorCount.toString());
auditFileWriter.newLine();
auditFileWriter.close();
}
}
//Main Process loop for the Delete Folders process.
//Outer loop executes the retrieval of Folders from the source Object Store Events until no matches are returned.
//Inner loop processes each Folder Deletion Event returned from the retrieval.
// It also detects and processes Update Folder and Unfile Document Events
//ASB ... 3 003 deleteFolders Method
private void deleteFolders() throws Exception {
Id GUID = null;
BufferedWriter auditFileWriter;
Long deleteStartTime;
Long folderDeleteStartTime;
Long timeToDelete;
long maxRunTimeMillis;
boolean moreFoldersToDelete=true;
Integer consecutiveErrs=0;
Integer errorCount=0;
boolean completed=false;
processedCount = 0;
updateSecurityProcessedCount = 0;
deleteStartTime = System.currentTimeMillis();
maxRunTimeMillis = cemc.getImportMaxRunTimeMinutes()*60*1000;
Long maxFolderCount = new Long(cemc.getDeleteMaxFolderCount());
auditFileWriter = new BufferedWriter(new FileWriter(cemc.getDeleteFolderAuditFile()));
auditFileWriter.write("Delete Start - ");
auditFileWriter.newLine();
auditFileWriter.write("GUID, SOURCEVSID, DESTDOCID, STATE, Millisecs");
auditFileWriter.newLine();
//ASB...1 Should be already connected here
while (moreFoldersToDelete && completed==false)
{
/*
Query to use is as follows:
*/
moreFoldersToDelete=false;
SearchSQL sqlObject = new SearchSQL();
sqlObject.setSelectList("e.CmAuditSequence,e.ClassDescription,e.Creator,e.DateCreated,e.SourceObjectId,e.SourceObject,e.DateLastModified, e.EventStatus, e.Id, e.InitiatingUser, e.LastModifier, e.Name, e.Owner, e.SourceClassId");
//ASB ... 2 022
Integer maxRecords = cemc.getDeleteMaxFolderCount();
sqlObject.setMaxRecords(maxRecords); //was hard coded to 20 !
sqlObject.setFromClauseInitialValue("Event", "e", true);
sqlObject.setFromClauseAdditionalJoin(JoinOperator.INNER, "CLASSDEFINITION", "cd", "e.SourceClassId", JoinComparison.EQUAL, "cd.Id", true);
//ASB ... 022 - Retrieve Date start from config.xml
String sSearchDate = cemc.getStartSearchDate();
//ASB ... 022 - enter date to search
//ASB ... 4 017 Search for just the Success Audit Events
sqlObject.setWhereClause("(e.DateCreated > " + sSearchDate.trim() + ") AND (" + folderSymbolList + ") AND (e.EventStatus = 0)" );
sqlObject.setOrderByClause("e.CmAuditSequence,e.DateCreated");
// Check the SQL statement.
logger.info("SQL: " + sqlObject.toString());
// Create a SearchScope instance. (Assumes you have the object store
// object.)
SearchScope search = new SearchScope(osinput);
// Set the page size (Long) to use for a page of query result data. This value is passed
// in the pageSize parameter. If null, this defaults to the value of
// ServerCacheConfiguration.QueryPageDefaultSize.
Integer myPageSize = new Integer(1000);
// Specify a property filter to use for the filter parameter, if needed.
// This can be null if you are not filtering properties.
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 1;
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
// Set the (Boolean) value for the continuable parameter. This indicates
// whether to iterate requests for subsequent pages of result data when the end of the
// first page of results is reached. If null or false, only a single page of results is
// returned.
Boolean continuable = new Boolean(true);
// Execute the fetchObjects method using the specified parameters.
IndependentObjectSet myObjects = search.fetchObjects(sqlObject, myPageSize, myFilter, continuable);
// You can then iterate through the collection of rows to access the properties.
int rowCount = 0;
Iterator iter = myObjects.iterator();
EngineObject unFileDoc = null;
Long lastFileEvent = null;
int lastEventSequence = 0;
while (iter.hasNext() && completed==false){
folderDeleteStartTime = System.currentTimeMillis();
Boolean deleteFailure = false;
IndependentObject object = (IndependentObject) iter.next();
Properties props = object.getProperties();
String eventDesc = "unknown"; //ASB ... 3 004 Set to Event Type default
Iterator prop = props.iterator();
int eventSequence = 0;
while (prop.hasNext()){
Property eventProp = (Property) prop.next();
//ASB ... 3 004 Get Event Type
if (eventProp.getPropertyName().equalsIgnoreCase("ClassDescription")){
eventDesc = eventProp.getEngineObjectValue().getProperties().get("SymbolicName").getStringValue();
}
//ASB ... 3 003 Check for the Folder ID property to delete
if (eventProp.getPropertyName().equalsIgnoreCase("SourceObjectId")){
GUID = eventProp.getIdValue();
}
//ASB ... 3 004 Get The Source Object (for unfile)
if (eventProp.getPropertyName().equalsIgnoreCase("SourceObject")){
unFileDoc = eventProp.getEngineObjectValue();
}
if (eventProp.getPropertyName().equalsIgnoreCase("CmAuditSequence")){
eventSequence = eventProp.getFloat64Value().intValue();
}
} //prop.hasNet()
String guid = GUID.toString().trim();
try {
//ASB ... 3 004 Check The Event Type is a DeletionEvent
if (eventDesc.equalsIgnoreCase("DeletionEvent")){
//ASB ... 3 003 Delete the Target Folder with the Delete event in the Source
deleteFolder(GUID.toString());
processedCount++;
}
//ASB ... 3 004 Check The Event Type is a UpdateEvent
if (eventDesc.equalsIgnoreCase("UpdateEvent")){
//Update of the Folder Detected - copy the new Source Folder properties
updateFolder(GUID);
updateProcessedCount++;
}
//ASB ... 4 021 Check Folder Security Update Event
if (eventDesc.equalsIgnoreCase("UpdateSecurityEvent")){
//Update of the Folder Detected - copy the new Source Folder properties
updateFolder(GUID);
updateSecurityProcessedCount++;
}
//ASB ... 3 004 Check The Event Type is a UnFile
if (eventDesc.equalsIgnoreCase("UnfileEvent")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null)); //ASB ... 4 020 added here
Folder f = Factory.Folder.fetchInstance(os, GUID, myFilter);
//Fetch Document Object DRCR Object
// Dynamic Referential Containment Relationship
//ASB ... 4 032 Detect a File/Unfile event and ignore unfile here!!
if (unFileDoc != null) {
EngineObject sourceObject = unFileDoc;
String objClass = sourceObject.getClassName();
if(objClass.equalsIgnoreCase("DynamicReferentialContainmentRelationship")){
com.filenet.api.core.DynamicReferentialContainmentRelationship DRCR = (com.filenet.api.core.DynamicReferentialContainmentRelationship) sourceObject;
Document doc = (Document) DRCR.get_Head();
Id removeGUID = doc.get_Id();//Get Source Doc
Document drem = Factory.Document.fetchInstance(os, removeGUID, myFilter);
com.filenet.api.core.ReferentialContainmentRelationship RCR_returned = f.unfile(drem);
RCR_returned.save(RefreshMode.REFRESH);
unfileProcessedCount++;
}
}
}
//ASB ... 4 030 Check The Event Type is a File Event
if (eventDesc.equalsIgnoreCase("FileEvent")){
lastEventSequence = eventSequence;
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null)); //ASB ... 4 020 added here
Folder f = Factory.Folder.fetchInstance(os, GUID, myFilter);
//Fetch Document Object DRCR Object
// Dynamic Referential Containment Relationship
if (unFileDoc != null){
EngineObject sourceObject = unFileDoc;
String objClass = sourceObject.getClassName();
if(objClass.equalsIgnoreCase("DynamicReferentialContainmentRelationship")){
com.filenet.api.core.DynamicReferentialContainmentRelationship DRCRsource = (com.filenet.api.core.DynamicReferentialContainmentRelationship) sourceObject;
Document DOCsource = (Document) DRCRsource.get_Head();
Id DRCR_id = DRCRsource.get_Id();
Id addGUID = DOCsource.get_Id();//Get Source Doc;
//Check if document is not already linked
Document drem = Factory.Document.fetchInstance(os, addGUID, myFilter);
//Unfile Document First!!
try{
com.filenet.api.core.ReferentialContainmentRelationship RCR_returned = f.unfile(drem);
RCR_returned.save(RefreshMode.REFRESH);
}catch(Exception eUnfile){
}
com.filenet.api.core.DynamicReferentialContainmentRelationship DRCR = Factory.DynamicReferentialContainmentRelationship.createInstance(os, ClassNames.DYNAMIC_REFERENTIAL_CONTAINMENT_RELATIONSHIP, DRCR_id);
DRCR.set_Head(drem);
DRCR.set_Tail(f);
DRCR.save(RefreshMode.NO_REFRESH);
fileProcessedCount++;
}
}
}
}
//catch (CEMigrateException ceme){
catch (Exception ceme){
try { // getting occasional unexpected errors on the server, try to make sure doc is deleted
if (eventDesc.equalsIgnoreCase("DeletionEvent")){
//Protect from other exceptions which might Occur!!
String folderId = GUID.toString();
deleteFolder(folderId);
//ASB ... 2 013 - Set Import failure to true here
deleteFailure = true;
}
}catch (Exception e){
//Nothing to throw here
}
consecutiveErrs ++;
errorCount++;
logger.warn(ceme.getMessage());
if (consecutiveErrs > cemc.getDeleteMaxErrorCount())
{
auditFileWriter.write("Maximum number of consecutive delete errors reached - Exiting.");
auditFileWriter.newLine();
logger.info("Maximum number of consecutive errors reached - Exiting.");
throw ceme;
}
}
}
//} Moved While
auditFileWriter.write("Finished - time taken " + (System.currentTimeMillis() - deleteStartTime) + "milliseconds");
auditFileWriter.newLine();
auditFileWriter.write("Finished - folders deleted - " + String.valueOf(processedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - folders updated - " + String.valueOf(updateProcessedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - folder security updated - " + String.valueOf(updateSecurityProcessedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - Documents unfiled - " + String.valueOf(unfileProcessedCount));
auditFileWriter.newLine();
auditFileWriter.write("Finished - Documents filed - " + String.valueOf(fileProcessedCount)); //ASB ... 4 030
auditFileWriter.newLine();
auditFileWriter.write("Finished - folders failed to be deleted - " + errorCount.toString());
auditFileWriter.newLine();
auditFileWriter.close();
}
}
//ASB ... 3 004 Update the Target Folder ACL and properties from the updated source Folder
private void updateFolder(Id GUID) throws Exception {
Folder parent = os.get_RootFolder(); //SET AS DEFAULT
int myFilterLevel = 1; //Changed from 4 to 1 for performance!
PropertyFilter myFilter = new PropertyFilter();
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_LIST, null));
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
//myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Folder f = Factory.Folder.fetchInstance(os, GUID, myFilter);
try {
//Get the Current Source Folder Object referenced in the Event table
Folder currentFolder = Factory.Folder.fetchInstance(osinput, GUID, myFilter);
//Fetch The Target Folder requiring modification
Date dateLastModified = currentFolder.get_DateLastModified();
String sModifier = currentFolder.get_LastModifier();
String sOwner = currentFolder.get_Owner();
String path = currentFolder.get_PathName();
//Get Target Object Store Folder's Parent Folder
String pathParts[] = path.split("/");
if (pathParts.length > 2){
String parentname = path.substring(0, path.lastIndexOf("/"));
parent = (Folder)os.getObject("Folder", parentname);
}
String folderclass = currentFolder.getClassName();
String name = currentFolder.get_Name();
//Update the properties and security from the Source Object Folder properties and ACL
f.set_Parent(parent);
f.set_FolderName(name);
f.set_DateLastModified(dateLastModified);
f.set_LastModifier(sModifier);
f.set_Owner(sOwner);
AccessPermissionList aclIn = currentFolder.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
f.set_Permissions(aclIn);
f.save(RefreshMode.REFRESH);
}
catch (Exception e){
logger.error(e.getMessage(), e);
}
}
//ASB ... 4 011 Update the Target Document ACL and properties from the updated source Document
private void updateDocument(Id GUID) {
//ASB ... 2 007 Fetch the Source Object Document to be Updated
try {
PropertyFilter myFilter = new PropertyFilter();
//ASB ... 4 Changed from 2 to 1 was 12 seconds for 3 updates
// now 8.5 Seconds for 3 updtes
int myFilterLevel = 1; //Changed from 4 to 2 -- TODO Set as a parameter
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Document d = Factory.Document.fetchInstance(os, GUID, myFilter);
//Get the Current Source Document Object referenced in the Event table
Document currentDocument = Factory.Document.fetchInstance(osinput, GUID, myFilter);
//ASB ... 4 020 Check if this is a reserved Document Object, if so Don't process!!
if(currentDocument.get_IsReserved()){
return;
}
//Fetch The Target document requiring modification
Properties props = currentDocument.getProperties(); //TODO ASB ... 3
updateDocProps(d, props, currentDocument);
//Update the properties and security from the Source Object Document properties and ACL
AccessPermissionList aclIn = currentDocument.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
d.set_Permissions(aclIn);
d.save(RefreshMode.REFRESH);
}
catch (Exception e){
logger.error(e.getMessage(), e);
}
}
//This method returns the Document versions in ascending order
//(Unfortunately FileNet API iterator returns latest version first! )
private Document[][] getDocumentVersionsArray(VersionSeries versions, Document [][] docVersions,int maxMin){
int versionCount=0;
VersionableSet verSet = versions.get_Versions();
Iterator versIt = verSet.iterator();
int versCount = 0;
Document doc = null; //Input Document versions
maxMin = 0;
int min=0;
int maj=0;
while (versIt.hasNext()){
versCount ++;
doc = (Document)versIt.next();
maj = doc.get_MajorVersionNumber();
min = doc.get_MinorVersionNumber();
//Record largest minor version value for loop
if (min > maxMin){
maxMin = min;
}
docVersions[maj][min] = doc;
}
versionCount = versCount;
return docVersions;
}
//This method updates the security ACLs from a source Object
//to a target Object
private Boolean upDateSecurityACLSet(IndependentlyPersistableObject ipoInput, IndependentlyPersistableObject ipoOutput){
Boolean changedACLs = false;
Properties propsIn = ipoInput.getProperties();
Properties propsOut = ipoOutput.getProperties();
//Check to ensure we have ACL properties to update
if (!propsIn.isPropertyPresent(PropertyNames.PERMISSIONS))
{
// No ACLs so return false
return changedACLs;
}
if (!propsOut.isPropertyPresent(PropertyNames.PERMISSIONS))
{
// No ACLs so return false
return changedACLs;
}
//Get source ACL list
AccessPermissionList aclIn =(AccessPermissionList)propsIn.getDependentObjectListValue(PropertyNames.PERMISSIONS);
AccessPermissionList aclOut =(AccessPermissionList)propsOut.getDependentObjectListValue(PropertyNames.PERMISSIONS);
//Replace ACL list in target object
//Remove existing ACLs
Iterator iterOut = aclOut.iterator();
while (iterOut.hasNext())
{
AccessPermission apIn = (AccessPermission)iterOut.next();
aclOut.remove(apIn);
}
//Add Source ACLs
Iterator iterIn = aclIn.iterator();
while (iterIn.hasNext())
{
AccessPermission apIn = (AccessPermission)iterIn.next();
aclOut.add(apIn);
}
//Set in target Object
changedACLs = true;
return changedACLs;
}
//Changes all groups and user ACLs
//To 'read-only' except the passed User and Group for a Document or Folder access permission list.
//User access is limited by altering the ACL permission
private AccessPermissionList addPermissions(AccessPermissionList apl, String excludeGroupName, String excludeUserName)
{
//ASB ... 4 007 Changed to exclude a user and a group
//ASB ... 2 021 - Property Security Filter
PropertyFilter PropF = new PropertyFilter();
PropF.addIncludeProperty(new FilterElement(null, null, null, PropertyNames.ID, null));
//Save the original APL
AccessPermissionList aplSaved = apl;
AccessPermissionList aplWorking = Factory.AccessPermission.createList();
Boolean excludeGranteeFound = false; //Set true if at least one APL is reserved
//Create a new access permission object.
//com.filenet.api.security.AccessPermission ap = Factory.AccessPermission.createInstance();
int PropFLevel = 1; // TODO Set as a parameter
PropF.setMaxRecursion(PropFLevel);
PropF.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
//Get Security Realm
com.filenet.api.security.Realm realm = Factory.Realm.fetchCurrent(os.getConnection(), PropF);
GroupSet g = realm.findGroups(excludeGroupName,
PrincipalSearchType.EXACT, //search Type Needed
PrincipalSearchAttribute.DISPLAY_NAME, //searchAttribute Needed
null, //Sort Type
100, //Page Size
PropF);
Iterator iterG = g.iterator();
UserSet u = realm.findUsers(excludeUserName,
PrincipalSearchType.EXACT, //search Type Needed
PrincipalSearchAttribute.DISPLAY_NAME, //searchAttribute Needed
null, //Sort Type
100, //Page Size
PropF);
Iterator iterU = u.iterator();
logger.info("Realm Name: " + realm.get_Name()); //print the name of the realm for the current user
//Set Read Only Access Bits
final int ACCESS_READONLY = AccessRight.READ.getValue() |
AccessRight.VIEW_CONTENT.getValue() |
AccessRight.CONNECT.getValue();
Iterator iterAp = apl.iterator();
//Go through each ACE in turn setting the required permissions
while (iterAp.hasNext())
{
com.filenet.api.security.AccessPermission apWorking = Factory.AccessPermission.createInstance();
com.filenet.api.security.AccessPermission ap = (com.filenet.api.security.AccessPermission) iterAp.next();
//Set the returned permission
// Get Grantee Name
Boolean groupFound = false;
Boolean userFound = false;
String sGrantee = ap.get_GranteeName();
iterG = g.iterator();
while (iterG != null && iterG.hasNext() == true)
{
Group group = (Group) iterG.next();
String sGroupDisplay = group.get_DisplayName();
String sGroupShortName = group.get_ShortName();
String sGroupName = group.get_Name();
String sGroupDistinguishedName = group.get_DistinguishedName();
// Set permissions based on whether Group is in the Read Only Group.
if (group.get_DisplayName().equalsIgnoreCase(sGrantee)
||group.get_Name().equalsIgnoreCase(sGrantee)
||group.get_ShortName().equalsIgnoreCase(sGrantee)
||group.get_DistinguishedName().equalsIgnoreCase(sGrantee) )
{
groupFound = true;
excludeGranteeFound = true;
} //Group Search in apl
}
iterU = u.iterator();
while (iterU != null && iterU.hasNext() == true)
{
com.filenet.api.security.User user = (com.filenet.api.security.User) iterU.next();
String sUserDisplay = user.get_DisplayName();
String sUserShortName = user.get_ShortName();
String sUserName = user.get_Name();
String sUserDistinguishedName = user.get_DistinguishedName();
// Set permissions based on whether Group is in the Read Only Group.
if (sUserDisplay.equalsIgnoreCase(sGrantee)
||sUserName.equalsIgnoreCase(sGrantee)
||sUserShortName.equalsIgnoreCase(sGrantee)
||sUserDistinguishedName.equalsIgnoreCase(sGrantee))
{
userFound = true;
excludeGranteeFound = true;
} //UserSearch in apl
}
// Set Read Only access permissions if the user and group are not excluded.
if(!(groupFound || userFound) ){
apWorking.set_GranteeName(sGrantee);
apWorking.set_AccessType(AccessType.ALLOW);
apWorking.set_AccessMask(new Integer(ACCESS_READONLY));
aplWorking.add(apWorking);
} else{
aplWorking.add(ap);
}
}
//Set and save the new permissions.
// Check if we have rescued at least one ACE as excluded
if (!excludeGranteeFound){
apl = aplSaved;
} else{
apl = aplWorking;
}
return apl;
}
//Changes all groups and user ACLs
//To 'read-only' except the passed User and Group for a Document or Folder access permission list.
//User access is limited by altering the ACL permission
private AccessPermissionList addPermissionsNoSearch(AccessPermissionList apl, String excludeGroupName, String excludeUserName)
{
//ASB ... 4 007 Changed to exclude a user and a group
//ASB ... 2 021 - Property Security Filter
PropertyFilter PropF = new PropertyFilter();
PropF.addIncludeProperty(new FilterElement(null, null, null, PropertyNames.ID, null));
//Save the original APL
AccessPermissionList aplSaved = apl;
AccessPermissionList aplWorking = Factory.AccessPermission.createList();
Boolean excludeGranteeFound = false; //Set true if at least one APL is reserved
//Create a new access permission object.
//com.filenet.api.security.AccessPermission ap = Factory.AccessPermission.createInstance();
int PropFLevel = 1; // TODO Set as a parameter
PropF.setMaxRecursion(PropFLevel);
PropF.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
//Set Read Only Access Bits
final int ACCESS_READONLY = AccessRight.READ.getValue() |
AccessRight.VIEW_CONTENT.getValue() |
AccessRight.CONNECT.getValue();
Iterator iterAp = apl.iterator();
//Go through each ACE in turn setting the required permissions
while (iterAp.hasNext())
{
com.filenet.api.security.AccessPermission apWorking = Factory.AccessPermission.createInstance();
com.filenet.api.security.AccessPermission ap = (com.filenet.api.security.AccessPermission) iterAp.next();
//Set the returned permission
// Get Grantee Name
Boolean groupFound = false;
Boolean userFound = false;
excludeGranteeFound = false;
String sGrantee = ap.get_GranteeName();
if (excludeGroupName.equalsIgnoreCase(sGrantee)){
groupFound = true;
excludeGranteeFound = true;
if(cemc.getDebugOutputFlag().equalsIgnoreCase("on")){
logger.info("Group Excluded : " + sGrantee); //print the name of the Group for the current user
}
}
if (excludeUserName.equalsIgnoreCase(sGrantee)){
userFound = true;
excludeGranteeFound = true;
if(cemc.getDebugOutputFlag().equalsIgnoreCase("on")){
logger.info("User Excluded : " + sGrantee); //print the name of the User for the current user
}
}
// Set Read Only access permissions if the user and group are not excluded.
if(!(groupFound || userFound) ){
apWorking.set_GranteeName(sGrantee);
apWorking.set_AccessType(AccessType.ALLOW);
apWorking.set_AccessMask(new Integer(ACCESS_READONLY));
aplWorking.add(apWorking);
if(cemc.getDebugOutputFlag().equalsIgnoreCase("on")){
logger.info("Group/User set with Read Only : " + sGrantee); //print the name of the Group for the current user
}
} else{
if(cemc.getDebugOutputFlag().equalsIgnoreCase("on")){
logger.info("User or Group Excluded from Read Only : " + sGrantee); //print the name of the User for the current user
}
aplWorking.add(ap);
}
}
//Set and save the new permissions.
// Check if we have rescued at least one ACE as excluded
if (!excludeGranteeFound){
apl = aplSaved;
} else{
apl = aplWorking;
}
return apl;
}
private Document createVersions(String documentTitle, Id GUID, IndependentObject object, Properties props, Iterator iterProps,Boolean isCurrent, Boolean isReserved,Document[][] docVersions,Integer majCount,int minCount, BufferedWriter auditFileWriter)throws Exception{
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 2; //Changed from 4 to 2 -- TODO Set as a parameter
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Document currentDocument = (Document) object;
String documentclass = currentDocument.getClassName();
Document dInput = Factory.Document.fetchInstance(osinput, GUID, myFilter);
//Create Version Series for the document in the Target Object Store
VersionSeries versions = dInput.get_VersionSeries();
//Get the Document Versions to be added from the Source Document
VersionableSet verSet = versions.get_Versions();
Iterator versIt = verSet.iterator();
int versCount = 0;
int majorVn = 0;
int minorVn = 0;
Document doc = null; //Input Document versions
Document d = null; //Output document versions
isReserved = false;
Id createdGuid = null;
Id nextGUID = null;
int foldCount = 0;
//while (versIt.hasNext()){ ASB ... 2
int minVersCount = -1;
while (versCount < majCount){
versCount ++;
while (minVersCount < minCount){
minVersCount ++;
doc = docVersions[versCount][minVersCount];
if(doc == null){
break;
}
foldCount ++;
majorVn = doc.get_MajorVersionNumber();
minorVn = doc.get_MinorVersionNumber();
nextGUID = doc.get_Id();
//ASB ... 4 004 Get the Release Version
VersionStatus dVersionStatus = doc.get_VersionStatus();
if (!isReserved)isReserved = doc.get_IsReserved();
//Need to update for each version in turn
d = createVersion(versCount, d, doc, isReserved, majorVn,minorVn, documentTitle, nextGUID, createdGuid, nextGUID,auditFileWriter);
createdGuid = d.get_Id(); //ASB ... 4 025 Try/Catch Here especially as reservation has no folders!!
try {
if (foldCount == majCount){
FolderSet fs = doc.get_FoldersFiledIn();
Iterator iterFs = fs.iterator();
while (iterFs.hasNext())
{
Folder folder = (Folder)iterFs.next();
//Create/Check and Make link in Target for the Document to this Folder
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("
Folder Name: " + folder.get_FolderName() +
" Folder Path: " + folder.get_PathName());
}
folderDoc(d,folder.get_PathName(),documentTitle);
}
}
}catch(Exception FileErr){
logger.error(FileErr.getMessage(), FileErr);
}
if (isReserved){
break;
}
}
//ASB ... 4 019 Check at this point if we have reached the completed creation
minVersCount = -1;
} // Next Major Version
if (isReserved)
d = doReservationProperties(versCount, doc, d, isReserved, majorVn,minorVn,documentTitle);
return d;
} //This method adds the content specified in the XML Nodes passed in to the specified document object.
//Creates a Content Transfer object for each node. Sets RetrievalName to the original filename
//to ensure that when the version is downloaded it as the correct file name Sets mimetype.
private void addContent(Document dSource, Document d, String documentTitle, PropertyFilter myFilterId, Id GUID, String DocClass) throws Exception {
//ADD CONTENT NEW CODE HERE
//ASB ... 4 024
int retryCount = 0;
Document currDoc = null;
while (retryCount < retryLimit){
try {
currDoc = (Document)osinput.fetchObject(DocClass, GUID, myFilterId);
//ASB ... 4 031 Add retry 7 times to fix API bug!!
ContentTransfer ct = Factory.ContentTransfer.createInstance();
InputStream str = currDoc.accessContentStream(0);
ct.setCaptureSource(str);
// Add Document Title
ct.set_RetrievalName(documentTitle);
// Add Content Mime Type
ct.set_ContentType(currDoc.get_MimeType());
ContentElementList cel = Factory.ContentElement.createList();
cel.add(ct);
d.set_ContentElements(cel);
break;
}
catch (Exception e){
String testMessage = e.getMessage();
String testCode = e.toString();
if( testMessage.contains("A uniqueness requirement has been violated")){
retryCount = retryLimit;
break;
}
//ASB ... 4 Check if the Content could not be added because the document already exists, if so
// Remove document and retry (next run)
// ASB ... 4 025 Log Exception here
logger.error(e.getMessage(), e);
retryCount ++;
retryTotalProcessedCount ++;
logger.info("Retry Count - createVersion method : " + retryCount + " of " + retryLimit + " On : " + documentTitle + " : with GUID : " + GUID);
}
} //ASB ... 4 031 Above loops Test retryCount less than the retryLimit of 7
// Add Security ACL
//ASB ... 2 020 Update Document ACL from source Document security
AccessPermissionList aclIn = currDoc.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//ASB ... 4 027 aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
d.set_Permissions(aclIn);
Boolean isMajorVersion = false;
if(currDoc.get_MinorVersionNumber() == 0) isMajorVersion = true;
//ASB ... 4 028 Change to do not Classify from AutoClassify.AUTO_CLASSIFY, to
if (isMajorVersion){
d.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MAJOR_VERSION);
}
else {
d.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MINOR_VERSION);
}
//d.checkin(AutoClassify.AUTO_CLASSIFY, CheckinType.MAJOR_VERSION);
//ASB ... 4 25 Try/Catch here ?
Date dateLastModified = dSource.get_DateLastModified();
//Date dateCreated = dSource.get_DateCreated(); //ASB ... 4 029
String sModifier = dSource.get_LastModifier();
Date dateCheckedIn = dSource.get_DateCheckedIn();
// Update Checkin Date etc from Source here
//d.set_DateCreated(dateCreated); //ASB ... 4 029 READONLY AT THIS POINT!!
d.set_DateLastModified(dateLastModified);
d.set_LastModifier(sModifier);
d.set_DateCheckedIn(dateCheckedIn); //ASB ... 2 005
d.save(RefreshMode.REFRESH); //ASB XXX
}
//Sets the properties on the reservation object for docs that were checked out on the source system.
//Sets properties but no content. No need to set mime type here – causes problems.
//private Document doReservationProperties(Node verNode, Document d) throws Exception {
private Document doReservationProperties(int versCount, Document dInput, Document d, Boolean isReserved, int majorVn,int minorVn, String documentTitle ) throws Exception{
Properties props = Factory.Document.createInstance(os, null).getProperties();
String sCreator = dInput.get_Creator();
Date dateCreated = dInput.get_DateCreated();
String sLastModifier = dInput.get_LastModifier();
Date dateLastModified = dInput.get_DateLastModified();
// This collection is used when we do the checkout
props.putValue("Creator", sCreator);
props.putValue("DateCreated", dateCreated);
props.putValue("LastModifier", sLastModifier);
props.putValue("DateLastModified", dateLastModified);
Date dateCheckedIn = dInput.get_DateCheckedIn();
//Update Checkin Date etc from Source here
d.set_LastModifier(sCreator);
d.set_DateLastModified(dateCreated);
d.set_DateCreated(dateCreated); //ASB ... 4 029
Id reservationId_null = null;
String reservationClass_null = null; // to remind of the parameter types!
try
{
// Get Document properties
Properties docProps = d.getProperties();
Iterator iterProps = props.iterator();
Boolean justCreated = false;
writeSpecialDocProps(d, iterProps, justCreated,documentTitle, dInput);
}
catch (Exception e){
throw new Exception(e.getMessage(), e);
}
d.set_LastModifier(sLastModifier);
// ASB Compare : d.checkout(ReservationType.EXCLUSIVE, GUID, docClass, null);
d.checkout(ReservationType.OBJECT_STORE_DEFAULT, reservationId_null, reservationClass_null, props);
d.set_DateLastModified(dateLastModified);
d.set_DateCreated(dateCreated); //ASB ... 4 029
d.set_LastModifier(sLastModifier);
d.save(RefreshMode.REFRESH);
return d;
}
//Called by CreateDocument, creates a version from the information in the XML node specified.
//Creates a new reservation object, specifying properties from XML.
//private Document createVersion(Document d) throws CEMigrateException, Exception {
private Document createVersion(int versCount, Document d, Document dInput, Boolean isReserved, int majorVn, int minorVn,String documentTitle, Id GUID, Id createdGUID, Id nextGUID,BufferedWriter auditFileWriter) throws Exception {
Document res = null;
Long docImportStartTime = System.currentTimeMillis();
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 2; //Changed from 4 to 2 -- For performance!
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
Document currDoc = null;
try
{
int thisMajorVersionNumber = majorVn;
int thisMinorVersionNumber = minorVn; //ASB ... 3 004
String docClass = dInput.getClassName();
//ASB ... 3 006 Now get the current source Doc Version Series ID
Id vId = dInput.get_VersionSeries().get_Id();
if (!isClassExist(docClass))
throw new Exception("Document class " + docClass + " Does not exist");
Id reservationId_null = null; String reservationClass_null = null; // to remind of the parameter types!
boolean isMajorVersion = false;
boolean justCreated = true;
if (d == null){
if (thisMajorVersionNumber > 0)
isMajorVersion = true;
if (thisMinorVersionNumber > 0)
isMajorVersion = false;
//ASB ... 2 015 Added GUID to ensure we get the same later
//ASB ... 3 006 New Version of the first document create to store Version series ID
//ASB ... 3 006 Many thanks to David Greenhouse for supplying the call :-)
res = Factory.Document.createInstance(os, docClass, GUID, vId, com.filenet.api.constants.ReservationType.EXCLUSIVE);
//res = (Document)os.createObject(docClass,GUID); //ASB ... 3 006 Removed Old Creation
//Added this to ensure that these values were correctly set on the initial version
String mimetype = dInput.get_MimeType();
String filename = dInput.get_Name();
Date dateCreated = dInput.get_DateCreated();
res.set_MimeType(mimetype);
res.set_DateCreated(dateCreated); //ASB ... 4 029
//ASB 04/04/2022 - Update Creator and last Modifier and dates
String sCreator = dInput.get_Creator();
res.set_Creator(sCreator); //ASB 04/04/2022
String sLastModifier = dInput.get_LastModifier();
res.set_LastModifier(sLastModifier); //ASB 04/04/2022
Date dateLastModified = dInput.get_DateLastModified();
res.set_DateLastModified(dateLastModified); //ASB 04/04/2022
currDoc = dInput;
Properties docprops = Factory.Document.createInstance(os, null).getProperties(); //????
Properties props = res.getProperties();
Iterator iterProps = props.iterator();
writeSpecialDocProps(res, iterProps, justCreated, documentTitle, currDoc);
//ASB ... 4 023 Add Count for first version
this.processedCount = this.processedCount +1;
String guid = GUID.toString().trim();
//Write to audit log
Long timeToImport = System.currentTimeMillis() - docImportStartTime;
auditFileWriter.write(guid + "," + vId.toString() + "," + guid + ",Imported," + timeToImport.toString());
auditFileWriter.newLine();
}
else {
justCreated = false;
//GUID here should be from next iteration set above
//ASB ... 2 016 Changed to osinput from os! try nextGUID
//currDoc = (Document)os.fetchObject("Document", createdGUID, myFilter);
//ASB ... 4 024 Update to support other Document Types
currDoc = (Document)os.fetchObject(docClass, createdGUID, myFilter);
Properties props = Factory.Document.createInstance(os, null).getProperties();
String sCreator = currDoc.get_Creator();
Date dateCreated = currDoc.get_DateCreated();
String sLastModifier = currDoc.get_LastModifier();
Date dateLastModified = currDoc.get_DateLastModified();
// This collection is used when we do the checkout
props.putValue("Creator", sCreator);
props.putValue("DateCreated", dateCreated);
props.putValue("LastModifier", sLastModifier);
props.putValue("DateLastModified", dateLastModified);
//ASB ... 4 024
//d = (Document)os.fetchObject(docClass, createdGUID, myFilter);
d = currDoc; //No need to fetch again!
//d = (Document)os.fetchObject("Document", createdGUID, myFilter);
//ASB ... 4 014 New Version needs to use the GUID passed in !!
String sNewGuid = GUID.toString();
// ASB Compare : d.checkout(ReservationType.EXCLUSIVE, GUID, docClass, null);
d.checkout(ReservationType.EXCLUSIVE, GUID, docClass, props); //ASB ... 4 029 Changed from null to props
//d.set_DateCreated(dateCreated); //ASB004 029 READONLY AT THIS POINT!!
d.save(RefreshMode.REFRESH);
this.processedCount = this.processedCount +1;
String guid = GUID.toString().trim();
//Write to audit log
Long timeToImport = System.currentTimeMillis() - docImportStartTime;
auditFileWriter.write(guid + "," + d.get_VersionSeries().get_Id().toString() + "," + d.get_Id().toString() + ",Imported," + timeToImport.toString());
auditFileWriter.newLine();
// Get the reservation object
res = (Document)d.get_Reservation();
res.set_LastModifier(sLastModifier); //ASB00 4
res.set_DateLastModified(dateLastModified); //ASB00 4 Uodate from dateCreated ??!!
//res.set_DateCreated(dateCreated); //ASB004 029 READ ONLY AT THIS POINT!!!
String mimetype = currDoc.get_MimeType();
props.putValue("MimeType", mimetype);
int mvn = majorVn;
int minvn = minorVn;
if (thisMajorVersionNumber > 0) isMajorVersion = true;
if (thisMinorVersionNumber > 0) isMajorVersion = false;
dInput = currDoc;
GUID = nextGUID; //Get Current Content from version
}
try {
addContent(dInput, res, documentTitle, myFilter, GUID,docClass);
}
catch (Exception e){
//ASB ... 4 Check if the Content could not be added because the document already exists, if so
// Remove document and retry (next run)
// ASB ... 4 025 Log Exception here
logger.error(e.getMessage(), e);
}
if(isReserved){ //ASB ... 2 012 Required For Checkin the Document must be Status of reserved!!
if (isMajorVersion){
//ASB ... 4 028 Changed from NULL to AutoClassify.DO_NOT_AUTO_CLASSIFY
res.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MAJOR_VERSION);
res.save(RefreshMode.NO_REFRESH);
}
else {
res.checkin(AutoClassify.DO_NOT_AUTO_CLASSIFY,CheckinType.MINOR_VERSION);
res.save(RefreshMode.NO_REFRESH);
}
}
}
catch (Exception e)
{
e.printStackTrace();
//TODO If caused by VSID issue attempt Delete Here
throw e;
}
return res;
}
//Adds properties such as LastModifier to the document properties.
//Called by doReservationProperties, createVersion
private void writeSpecialDocProps(Document d,Iterator iterProps, Boolean justCreated, String documentTitle, Document dInput) throws Exception {
//ASB ... 2 008 Updated for new property definition objects
//Retrieve the current property list to retrieve based on the Document Class definitions from the Object Store
com.filenet.api.admin.ClassDefinition classDefs = Factory.ClassDefinition.fetchInstance(osinput,d.getClassName(), null);
PropertyDefinitionList propList = classDefs.get_PropertyDefinitions();
StringBuffer tempBuf = new StringBuffer();
//Retrieve property names
Iterator iter = propList.iterator();
while (iter.hasNext())
{
//ASB ... 4 025 Catch any property exceptions here
try{
PropertyDefinition propDef = (PropertyDefinition) iter.next();
String propName = propDef.get_SymbolicName();
//ASB ... 2 011 Need to skip Date Last Accessed etc - ReadOnly error here
if (!(propName.equalsIgnoreCase("DateContentLastAccessed")
||propName.equalsIgnoreCase("LockOwner")
||propName.equalsIgnoreCase("Name")
||propName.equalsIgnoreCase("StorageLocation")
||propName.equalsIgnoreCase("ContentElementsPresent")
||propName.equalsIgnoreCase("DateCheckedIn")
||propName.equalsIgnoreCase("ContentRetentionDate")
||propName.equalsIgnoreCase("CurrentState") )){
//ASB ... 2 011 Set to check cardinality
int CardinalityVal = propDef.get_Cardinality().getValue();
switch (propDef.get_DataType().getValue()){
case StringType:
if (CardinalityVal == CardinalitySINGLE){
String val = dInput.getProperties().getStringValue(propName);
d.getProperties().putValue(propName, val);
if (propName.equalsIgnoreCase("lastmodifier")){
d.set_LastModifier(val);
}
if (!propName.equalsIgnoreCase("creator") && !propName.equalsIgnoreCase("lastmodifier")){
d.getProperties().putValue(propName, val);
}
}else{
//List
StringList valList = dInput.getProperties().getStringListValue(propName);
d.getProperties().putValue(propName, valList);
}
break;
case DateType:
if (CardinalityVal == CardinalitySINGLE){
Date dateVal = dInput.getProperties().getDateTimeValue(propName);
if (propName.equalsIgnoreCase("datelastmodified")){
if (!justCreated){
dInput.getProperties().removeFromCache("DateLastModified");
}
d.set_DateLastModified(dateVal);
}
if (!propName.equalsIgnoreCase("datecreated") && !propName.equalsIgnoreCase("datelastmodified")){
d.getProperties().putValue(propName, dateVal);
}
}
//TODO Process Multi-value Dates (if exists)?
break;
default:
}
}
}catch(Exception writeProp){
logger.error(writeProp.getMessage(), writeProp);
}
}
//ASB ... 2 020 Update Document ACL from source Document security
AccessPermissionList aclIn = dInput.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//ASB ... 4 027 aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
d.set_Permissions(aclIn);
}
//Called by doReservationProperties, createVersion. Adds document properties.
private void writeDocProps(Document d,Properties props,Document currentDoc) throws Exception {
//Retrieve the current property list to retrieve based on the Document Class definitions from the Object Store
com.filenet.api.admin.ClassDefinition classDefs = Factory.ClassDefinition.fetchInstance(osinput,d.getClassName(), null);
PropertyDefinitionList propList = classDefs.get_PropertyDefinitions();
StringBuffer tempBuf = new StringBuffer();
//Retrieve property names
Iterator iter = propList.iterator();
while (iter.hasNext())
{
try {
PropertyDefinition propDef = (PropertyDefinition) iter.next();
//PropertyDefinition propDef = (PropertyDefinition) iter.next();
String propName = propDef.get_SymbolicName();
//ASB ... 2 011 Need to skip Date Last Accessed etc - ReadOnly error here
if (!(propName.equalsIgnoreCase("DateContentLastAccessed")
||propName.equalsIgnoreCase("LockOwner")
||propName.equalsIgnoreCase("Name")
||propName.equalsIgnoreCase("StorageLocation")
||propName.equalsIgnoreCase("ContentRetentionDate")
||propName.equalsIgnoreCase("ContentElementsPresent")
||propName.equalsIgnoreCase("DateCheckedIn")
||propName.equalsIgnoreCase("CurrentState") )){
//ASB ... 2 011 Set to check cardinality
int CardinalityVal = propDef.get_Cardinality().getValue();
switch (propDef.get_DataType().getValue()){
case StringType:
if (CardinalityVal == CardinalitySINGLE){
String val = currentDoc.getProperties().getStringValue(propName);
d.getProperties().putValue(propName, val);
if (propName.equalsIgnoreCase("lastmodifier")){
d.set_LastModifier(val);
}
if (!propName.equalsIgnoreCase("creator") && !propName.equalsIgnoreCase("lastmodifier")){
d.getProperties().putValue(propName, val);
}
}else{
//List
StringList valList = currentDoc.getProperties().getStringListValue(propName);
d.getProperties().putValue(propName, valList);
}
// Process Multi-value Strings (if required)?
break;
case DateType:
if (CardinalityVal == CardinalitySINGLE){
Date dateVal = currentDoc.getProperties().getDateTimeValue(propName);
if (propName.equalsIgnoreCase("datelastmodified")){
currentDoc.getProperties().removeFromCache("DateLastModified");
d.set_DateLastModified(dateVal);
}
//ASB ... 4 029 Start
if (propName.equalsIgnoreCase("datecreated")){
currentDoc.getProperties().removeFromCache("DateCreated");
d.set_DateLastModified(dateVal);
}
//ASB ... 4 029 End
if (!propName.equalsIgnoreCase("datecreated") && !propName.equalsIgnoreCase("datelastmodified")){
d.getProperties().putValue(propName, dateVal);
}
}
//TODO Process Multi-value Dates (if exists)?
break;
// }
default:
}
}
}catch(Exception writeProp){
logger.error(writeProp.getMessage(), writeProp);
}
} //for (int iProp = 0;iProp < xmlprops.getLength();iProp ++){
}
private void updateDocProps(Document d,Properties props,Document currentDoc) {
//Retrieve the current property list to retrieve based on the Document Class definitions from the Object Store
try {
PropertyFilter myFilter = new PropertyFilter();
//ASB ... 4 017 Changed from 2 to 1 for performance
// Changed from 8.5 seconds to 5.3 Seconds
int myFilterLevel = 1; //Changed from 4 to 1 -- TODO Set as a parameter
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_LIST, null));
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
//myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
String sDocClassName = d.getClassName();
com.filenet.api.admin.ClassDefinition classDefs = Factory.ClassDefinition.fetchInstance(osinput,sDocClassName, myFilter);
PropertyDefinitionList propList = classDefs.get_PropertyDefinitions();
StringBuffer tempBuf = new StringBuffer();
// ASB ... 4 Following has to be commented out because ct is readonly!
// Set Content Retrieval name - Can't because it is Read-Only
//ContentElementList ctElementList = d.get_ContentElements();
//Iterator iterCTList = ctElementList.iterator();
//ContentTransfer ct = null;
// Add Document Title
//Retrieve property names
Iterator iter = propList.iterator();
while (iter.hasNext())
{
try {
PropertyDefinition propDef = (PropertyDefinition) iter.next();
//PropertyDefinition propDef = (PropertyDefinition) iter.next();
String propName = propDef.get_SymbolicName();
//ASB ... 2 011 Need to skip Date Last Accessed etc - ReadOnly error here
if (!(propName.equalsIgnoreCase("DateContentLastAccessed")
||propName.equalsIgnoreCase("LockOwner")
||propName.equalsIgnoreCase("Name")
||propName.equalsIgnoreCase("Creator")
||propName.equalsIgnoreCase("StorageLocation")
||propName.equalsIgnoreCase("ContentRetentionDate")
||propName.equalsIgnoreCase("ContentElementsPresent")
||propName.equalsIgnoreCase("DateCheckedIn")
||propName.equalsIgnoreCase("MimeType")
||propName.equalsIgnoreCase("CurrentState") )){
//ASB ... 2 011 Set to check cardinality
int CardinalityVal = propDef.get_Cardinality().getValue();
switch (propDef.get_DataType().getValue()){
case StringType:
if (CardinalityVal == CardinalitySINGLE){
String val = currentDoc.getProperties().getStringValue(propName);
d.getProperties().putValue(propName, val);
if (propName.equalsIgnoreCase("lastmodifier")){
d.set_LastModifier(val);
}
if (!propName.equalsIgnoreCase("creator") && !propName.equalsIgnoreCase("lastmodifier")){
d.getProperties().putValue(propName, val);
}
if (propName.equalsIgnoreCase("DocumentTitle")){
/*
while (iterCTList.hasNext())
{
ct = (ContentTransfer) iterCTList.next();
ct.set_RetrievalName(val);
}
*/
}
}else{
//List
StringList valList = currentDoc.getProperties().getStringListValue(propName);
d.getProperties().putValue(propName, valList);
}
//TODO Process Multi-value Strings (if required)?
break;
case DateType:
if (CardinalityVal == CardinalitySINGLE){
Date dateVal = currentDoc.getProperties().getDateTimeValue(propName);
if (propName.equalsIgnoreCase("datelastmodified")){
currentDoc.getProperties().removeFromCache("DateLastModified");
d.set_DateLastModified(dateVal);
}
//ASB ... 4 029 Start
if (propName.equalsIgnoreCase("datecreated")){
currentDoc.getProperties().removeFromCache("DateCreated");
d.set_DateLastModified(dateVal);
}
//ASB ... 4 029 End
if (!propName.equalsIgnoreCase("datecreated") && !propName.equalsIgnoreCase("datelastmodified")){
d.getProperties().putValue(propName, dateVal);
}
}
//TODO Process Multi-value Dates (if exists)?
break;
// }
default:
}
}
}catch(Exception writeProp){
logger.error(writeProp.getMessage(), writeProp);
}
} //for (int iProp = 0;iProp < xmlprops.getLength();iProp ++){
} catch (Exception e){
logger.error(e.getMessage(), e);
}
}
//OLD Comment Called from main and imports Folders from Folders table.
//Called from main and imports Folders from Source Folders Event table.
private void importFolders() throws Exception {
BufferedWriter auditFileWriter;
long importStartTime;
long folderImportStartTime;
Long timeToImport;
// long maxRunTimeMillis;
long consecutiveErrs=0;
Long errorCount= new Long(0);
importStartTime = System.currentTimeMillis();
auditFileWriter = new BufferedWriter(new FileWriter(cemc.getImportAuditFileFolders()));
auditFileWriter.write("Import Start - ");
auditFileWriter.newLine();
auditFileWriter.write("GUID, PATH, STATE, EXPORTFILE, Millisecs");
auditFileWriter.newLine();
String OSName = cemc.getExportOSName().trim();
osinput = con_MigrateDB.getObjectStore(OSName);
os = ceConn.getObjectStore(cemc.getImportOSName());
ub = UpdatingBatch.createUpdatingBatchInstance(ceConn.getDomain(), RefreshMode.NO_REFRESH);
String exportFile = ""; //ASB Fix Imported Issue
String guid ="";
batchCount = 0; //ASB 24/02/2021 ADD
batchSize = cemc.getUpdatingBatchSize(); //ASB 24/02/2021 ADD
SearchSQL sqlObject = new SearchSQL();
sqlObject.setSelectList("f.foldername, f.Id, f.Creator, f.DateCreated, f.LastModifier, f.DateLastModified, f.Name, f.Owner, f.LockToken, f.LockTimeout, f.LockOwner, f.PathName, f.IndexationId, f.CmIndexingFailureCode, f.CmRetentionDate, f.ContainerType, f.InheritParentPermissions, f.IsHiddenContainer");
Integer maxRecords = cemc.getExportMaxDocCount();
sqlObject.setMaxRecords(maxRecords); //was hard coded to 20 !
sqlObject.setFromClauseInitialValue("Folder", "f", true); //Update to look for other Folder types
//Retrieve Date start from config.xml
String sSearchDate = cemc.getStartSearchDate();
//sqlObject.setWhereClause("f.DateCreated > " + "20200921T124517Z" );
sqlObject.setWhereClause("f.DateCreated > " + sSearchDate );
sqlObject.setOrderByClause("f.DateCreated");
// Check the SQL statement.
logger.info("SQL: " + sqlObject.toString());
// Create a SearchScope instance. (Assumes you have the object store
// object.)
SearchScope search = new SearchScope(osinput);
// Set the page size (Long) to use for a page of query result data. This value is passed
// in the pageSize parameter. If null, this defaults to the value of
// ServerCacheConfiguration.QueryPageDefaultSize.
Integer myPageSize = new Integer(1000);
// Specify a property filter to use for the filter parameter, if needed.
// This can be null if you are not filtering properties.
PropertyFilter myFilter = new PropertyFilter();
int myFilterLevel = 2; //Changed from 1 to 2 -- For performance !
myFilter.setMaxRecursion(myFilterLevel);
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
// Set the (Boolean) value for the continuable parameter. This indicates
// whether to iterate requests for subsequent pages of result data when the end of the
// first page of results is reached. If null or false, only a single page of results is
// returned.
Boolean continuable = new Boolean(true); //SET To allow more results to be fetched
// Execute the fetchObjects method using the specified parameters.
IndependentObjectSet myObjects = search.fetchObjects(sqlObject, myPageSize, myFilter, continuable);
// You can then iterate through the collection of rows to access the properties.
int rowCount = 0;
Iterator iter = myObjects.iterator();
while (iter.hasNext()){
try
{
folderImportStartTime = System.currentTimeMillis();
Boolean moreFoldersToImport=true;
Boolean importFailure = false;
Folder f = null;
IndependentObject object = (IndependentObject) iter.next();
//ASB Software Development Limited Fetch the full folder Object
//ASB Software Development Limited example found to illustrate full property retrieval 01-08-2022
// Id id = document.get_Id(); //get the doc ID back on this shallow document copy.
// *retrieve a complete document copy back with complete property set using doc ID*/
// Document docCopy = (Document)Factory.Document.fetchInstance(objectStore, id, null);
// FolderSet pFolders = doc.get_FoldersFiledIn();//Get the full "FolderFiledIn" property on the new copy
//
// ASB Properties props = object.getProperties();
//ASB Iterator iterProps = props.iterator();
Id GUID = ((Folder) object).get_Id();
//ASB Software Development Limited - get full document property set which we need! 01-08-2022
Folder folderCopy = (Folder)Factory.Folder.fetchInstance(osinput, GUID, null);
// ASB Properties props = object.getProperties();
Properties props = folderCopy.getProperties();
Iterator iterProps = props.iterator();
String folderName = "";
String folderPath = "";
guid = GUID.toString().trim();
while (iterProps.hasNext() )
{
com.filenet.api.property.Property prop = (com.filenet.api.property.Property)iterProps.next();
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
}else{
logger.info("
Property: " + prop.getPropertyName() );
if ( ((com.filenet.api.property.Property) prop).getObjectValue() != null )
logger.info(" Value: " + prop.getObjectValue().toString() );
}
if (prop.getPropertyName().equalsIgnoreCase("foldername"))
{
folderName = prop.getStringValue();
}
if (prop.getPropertyName().equalsIgnoreCase("PathName"))
{
folderPath = prop.getStringValue();
}
}
//ASB ... 3 005 Check Folder is in the path required
if (folderPath.startsWith( cemc.getImportOSRootFoler())){
//ASB f = createFolder(folderName,GUID,folderPath,object);
f = createFolder(folderName,GUID,folderPath,folderCopy);
}else{
//Folder selected is new but not in our import Folder Path!
logger.info("IGNORED: New Folder: " + folderName + " found linked to external Folder Path : " + folderPath + " Should be linked to :" + cemc.getImportOSRootFoler());
}
if (f != null){
//ASB 24/02/2022 New Code here
if (batchSize > 0)
ub.add(f,null);
doUpdateBatch(false);
processedCount++;
timeToImport = System.currentTimeMillis() - folderImportStartTime;
auditFileWriter.write(guid + "," + folderPath + ",Imported," + folderName + "," + timeToImport.toString());
auditFileWriter.newLine();
}
}
catch(Exception e)
{
consecutiveErrs ++;
errorCount++;
logger.warn(e.getMessage());
if (consecutiveErrs > cemc.getImportMaxErrorCount())
{
auditFileWriter.write("Maximum number of conecutive errors reached - Exiting.");
auditFileWriter.newLine();
logger.info("Maximum number of consecutive errors reached - Exiting.");
throw e;
}
}
}
doUpdateBatch(true);
auditFileWriter.write("Finished - time taken " + (System.currentTimeMillis() - importStartTime) + "milliseconds");
auditFileWriter.newLine();
auditFileWriter.write("Finished - folder processed - " + getProcessedCount());
auditFileWriter.newLine();
auditFileWriter.write("Finished - folder failed to be processed - " + errorCount.toString());
auditFileWriter.newLine();
auditFileWriter.close();
}
private void doUpdateBatch(boolean last){
if (batchSize > 0){
batchCount ++;
if (last || (batchCount >= batchSize)){
logger.debug("Commit batch");
try {
ub.updateBatch();
}
catch (Exception e){
Iterator iterator1 = ub.getBatchItemHandles(null).iterator();
while (iterator1.hasNext()) {
BatchItemHandle obj = (BatchItemHandle) iterator1.next();
if (obj.hasException())
{
// Displays the exception, for the purpose of brevity here.
EngineRuntimeException thrown = obj.getException();
logger.error("Exception: " + thrown.getMessage());
}
}
}
logger.debug("Commit batch OK");
batchCount = 0;
}
}
}
//Creates the folder defined by the XML passed in.
private Folder createFolder(String folderName, Id GUID,String folderPath, IndependentObject object) throws Exception {
Folder parent = os.get_RootFolder(); //SET AS DEFAULT
//Read from folder path from input Folder object (passed in)
String path = folderPath;
logger.debug("Importing Folder " + path); //ASB 24/02/2022
//ASB ... 2 020 Need to change to fetch for security ACL settings
Folder currentFolder = (Folder) object;
String folderclass = currentFolder.getClassName();
String name = folderName;
String sCreator = currentFolder.get_Creator();
Date dateCreated = currentFolder.get_DateCreated();
Date dateLastModified = currentFolder.get_DateLastModified();
String sModifier = currentFolder.get_LastModifier();
String sOwner = currentFolder.get_Owner();
String pathParts[] = path.split("/");
if (pathParts.length > 2){
// the parent should already exist, so just get it
String parentname = path.substring(0, path.lastIndexOf("/"));
parent = (Folder)os.getObject("Folder", parentname);
}
try {
//Create Folder with the same GUID as the Source Object
Folder f = Factory.Folder.createInstance(os, folderclass,GUID);
f.set_Parent(parent);
f.set_FolderName(name);
f.set_Creator(sCreator); //ASB 17/02/2022
f.set_DateCreated(dateCreated); //ASB 17/02/2022
f.set_DateLastModified(dateLastModified);
f.set_LastModifier(sModifier);
f.set_Owner(sOwner);
try {
//ASB ... 2 020 Update Folder ACL from source Folder security
int myFilterLevel = 1; //Changed from 4 to 1 -- For Performance
PropertyFilter myFilter = new PropertyFilter();
myFilter.setMaxRecursion(myFilterLevel);
//ASB ... 4 020
if(cemc.getDebugOutputFlag().equalsIgnoreCase("off")){
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_LIST, null));
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY_SINGLETON, null));
}else{
myFilter.addIncludeType(new FilterElement(null, null, null, FilteredPropertyType.ANY, null));
}
Folder currentFolderACL = Factory.Folder.fetchInstance(osinput, GUID, myFilter);
AccessPermissionList aclIn = currentFolderACL.get_Permissions();
// Add the permission to the list for the Object Store.
//Get This group from the config.xml file
String sExcludeGroup = cemc.getExcludedGroup();
String sExcludeUser = cemc.getExcludedUser();
//ASB ... 4 027 For Performance check if a full LDAP search is required
if(cemc.getLDAPSearchFlag().equalsIgnoreCase("on")){
aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
}else {
aclIn = addPermissionsNoSearch(aclIn, sExcludeGroup, sExcludeUser);
}
//ASB ... 4 027 aclIn = addPermissions(aclIn, sExcludeGroup, sExcludeUser);
f.set_Permissions(aclIn);
}catch(Exception aclErr){
//Just warning here if we can't set for some obscure reason!
logger.warn("Couldn't Add ACL Security for Folder " + aclErr.getMessage());
}
f.save(RefreshMode.REFRESH);
return f;
}
catch (Exception e){
logger.error(e.getMessage(), e);
}
return null;
}
private void init() throws Exception {
cemc = new CEReplicateConfig();
classNames = new HashMap();
logger.debug("Getting Destination (5x) CE Connection ...");
ceConn = new CEConnection("Target");
logger.debug("Dest (5x) CE Connection OK");
logger.debug("Getting Source (5x) CE Connection ...");
con_MigrateDB = new CEConnection("Source");
logger.debug("Migration Database Connection OK");
//ASB ... 4 022 Cater for Folder subclasses
String folderSubclasses = cemc.getFolderSubclasses();
String[] folderItems = folderSubclasses.split(",");
for (String item : folderItems)
{
//Add repeating units of
//" and (cd.SymbolicName = 'Folder') " etc
//(cd.SymbolicName = 'Folder') OR (cd.SymbolicName = 'FOLDER_ATTRIBUTE') etc
String nextFolder = item;
folderSymbolList = folderSymbolList + " (cd.SymbolicName = '" + nextFolder + "') ";
folderExcludeSymbolList = folderExcludeSymbolList + " AND (cd.SymbolicName <>'" + nextFolder + "') ";
folderSymbolList = folderSymbolList + "OR";
}
//Remove last OR
folderSymbolList = folderSymbolList.substring(1,folderSymbolList.length()-2 );
}
private void sampleCode() {
final String createUser = "CreateUser"; // we cant this on version 1
final String modUser = "ModUser"; // we cant this on version 2
// Get the ObjectStore
os = ceConn.getObjectStore("ECM");
// Create a new properties collection
Properties props = Factory.Document.createInstance(os, null).getProperties();
// This collection is used when we do the checkout
props.putValue("Creator", createUser);
//props.putValue("DateCreated", theDate);
props.putValue("LastModifier", modUser);
//props.putValue("DateLastModified", theDate);
// Folder here for easy find in Workplace
Folder f = (Folder)os.getObject("Folder", "/asb");
// Create the new document
Document d = (Document)os.createObject("Document");
d.getProperties().putValue("DocumentTitle", "Doc1234567890");
d.set_Creator(createUser);
d.set_LastModifier(createUser);
d.checkin(null, CheckinType.MAJOR_VERSION);
d.save(RefreshMode.NO_REFRESH); // If you refresh, you cant reset the last modifier
// Folder it (not needed, just makes it easier to find to see results)
if (ceContainmentNameExtOn.equalsIgnoreCase("yes")){
//ASB 01/03/2022 - Add Containment Name from the Title with no stripped extension
folderDocWithExtension(d, f);
}else if (ceContainmentNameExtOn.equalsIgnoreCase("no"))
{
//ASB 01/03/2022 - Add Containment Name from the Title with stripped extension
folderDoc(d,f);
}else{
// ASB 01/03/2022 - Add Containment Name using API Default
folderDocDefault(d,f);
}
// This makes sure the last modifier stays intact
d.set_LastModifier(createUser);
Id reservationId_null = null; String reservationClass_null = null; // to remind of the parameter types!
// ASB Compare : d.checkout(ReservationType.EXCLUSIVE, GUID, docClass, null);
d.checkout(ReservationType.EXCLUSIVE, reservationId_null, reservationClass_null, props);
// Have to do a save or the doc doesnt get checked out, needs to be refresh or we cant get the reservation
// object to check it back in
d.save(RefreshMode.REFRESH);
Document res = (Document)d.get_Reservation();
res.getProperties().removeFromCache("LastModifier");
res.set_LastModifier(modUser);
res.checkin(null,CheckinType.MAJOR_VERSION);
res.save(RefreshMode.NO_REFRESH);
//Encryption
try {
// Generate a temporary key. In practice, you would save this key.
// See also Encrypting with DES Using a Pass Phrase.
SecretKey key = KeyGenerator.getInstance("DES").generateKey();
// Encrypt
String encrypted = this.encrypt("Don't tell anybody!");
// Decrypt
String decrypted = this.decrypt(encrypted);
} catch (Exception e) {
}
}
private void folderDoc(Document doc, Folder folder){
//ASB 01/03/2022 - Add Containment Name from the Title with stripped extension
if (folder != null){
String sContainmentName = ""; //FOR RCR: //ASB 17/02/2022 Add Containment name
sContainmentName = doc.getProperties().get("DocumentTitle").getStringValue(); //FOR RCR: //ASB 17/02/2022 Add Containment name
ReferentialContainmentRelationship rcr = folder.file(doc,AutoUniqueName.AUTO_UNIQUE,null,DefineSecurityParentage.DO_NOT_DEFINE_SECURITY_PARENTAGE);
if(sContainmentName.lastIndexOf(".") > 0){ //FOR RCR: //ASB 17/02/2022 Add Containment name
sContainmentName = sContainmentName.substring(0,sContainmentName.lastIndexOf(".")); //FOR RCR: //ASB 17/02/2022 extract filename
} //FOR RCR: //ASB 17/02/2022 Add Containment name
if(!sContainmentName.equalsIgnoreCase("")){ //FOR RCR: //ASB 17/02/2022 Add Containment name
rcr.set_ContainmentName(sContainmentName); //FOR RCR: //ASB 17/02/2022 Add Containment name
} //FOR RCR: //ASB 17/02/2022 Add Containment name
rcr.save(RefreshMode.NO_REFRESH);
}
}
private void folderDoc(Document doc, String folderPathName, String documentTitle){
String sContainmentName = documentTitle; //FOR RCR: //ASB 17/02/2022 Add Containment name
if (folderPathName != null && folderPathName.length() > 0){
Folder folder = null;
try {
folder = Factory.Folder.getInstance(os, null, folderPathName);
//folder = Factory.Folder.fetchInstance(os, folderName, null);
com.filenet.api.core.DynamicReferentialContainmentRelationship rcr =
Factory.DynamicReferentialContainmentRelationship.createInstance(os, null,
AutoUniqueName.AUTO_UNIQUE,
DefineSecurityParentage.DO_NOT_DEFINE_SECURITY_PARENTAGE);
rcr.set_Tail(folder);
rcr.set_Head(doc);
rcr.set_ContainmentName(sContainmentName);
rcr.save(RefreshMode.NO_REFRESH);
}
catch (Exception e){
}
}
}
private void folderDocWithExtension(Document doc, Folder folder){
//ASB 01/03/2022 - Add Containment Name from the Title with no stripped extension
if (folder != null){
String sContainmentName = ""; //FOR RCR: //ASB 17/02/2022 Add Containment name
sContainmentName = doc.getProperties().get("DocumentTitle").getStringValue(); //FOR RCR: //ASB 17/02/2022 Add Containment name
ReferentialContainmentRelationship rcr = folder.file(doc,AutoUniqueName.AUTO_UNIQUE,null,DefineSecurityParentage.DO_NOT_DEFINE_SECURITY_PARENTAGE);
if(!sContainmentName.equalsIgnoreCase("")){ //FOR RCR: //ASB 17/02/2022 Add Containment name
rcr.set_ContainmentName(sContainmentName); //FOR RCR: //ASB 17/02/2022 Add Containment name
} //FOR RCR: //ASB 17/02/2022 Add Containment name
rcr.save(RefreshMode.NO_REFRESH);
}
}
private void folderDocDefault(Document doc, Folder folder){
//ASB 01/03/2022 - Add Containment Name using the API Default
if (folder != null){
ReferentialContainmentRelationship rcr = folder.file(doc,AutoUniqueName.NOT_AUTO_UNIQUE,null,DefineSecurityParentage.DO_NOT_DEFINE_SECURITY_PARENTAGE);
rcr.save(RefreshMode.NO_REFRESH);
}
}
private boolean isClassExist(String className) throws Exception {
if (classNames.containsKey(className))
return true;
try {
com.filenet.api.admin.ClassDefinition test = Factory.ClassDefinition.fetchInstance(os, className, null);
classNames.put(test.get_Name(), test.get_Name());
return true;
}
catch (EngineRuntimeException e){
if (e.getExceptionCode() == ExceptionCode.E_BAD_CLASSID)
return false;
throw e;
}
}
public static String encrypt(String str) {
try {
//encoding byte array into base 64
byte[] encoded = Base64.encodeBase64(str.getBytes());
String encString = new String(encoded);
return encString;
} catch (Exception e) {
}
return null;
}
public static String decrypt(String str) {
try {
//decoding byte array into base64
byte[] decoded = Base64.decodeBase64(str);
String sDecoded = new String(decoded);
return sDecoded;
} catch (Exception e) {
}
return null;
}
}
CEReplicateConfig – 502 Lines of Java Code –fn_common.jar
package com.ibm.ce.utils;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.StringWriter;
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import java.util.Vector;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;
import org.apache.commons.codec.binary.Base64;
import org.apache.log4j.Logger;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
public class CEReplicateConfig {
Logger logger = Logger.getLogger(CEReplicateConfig.class.getName());
private Document dom;
// Export Parameters
private String exportOSName;
private String exportCEUser;
private String exportCEPassword;
// private String exportSql;
private Integer exportMaxDocCount;
private Integer deleteMaxFolderCount;
private Integer exportMaxRunTimeMinutes;
private Integer exportMaxErrorCount;
private Integer deleteMaxErrorCount; //ASB ... 3 003 Max Count of deleted Document errors
//ASB ... 4 004 006 add Document propertyList values
private String exportAuditFile;
private Integer exportFeedBatchSize;
private Integer updateMaxErrorCount; //ASB ... 4 012 Max Count of updated Document errors
// Import parameters
private String importOSName;
private String importOSRootFolder;
private String importCEUrl;
private String importCELoginConfig; //ASB
private String importCEStanza;
private String importCEUser;
private String importCEPassword;
private Integer importMaxErrorCount;
private String importAuditFile;
//ASB ... 3 003 Set up Delete Document Audit Log file
private String deleteDocumentAuditFile;
//ASB ... 3 003 Set up Delete Folder Audit Log file
private String deleteFolderAuditFile;
//ASB ... 4 012 Set up Update Document Audit Log file
private String updateDocumentAuditFile;
private Integer importMaxDocCount;
private Integer deleteMaxDocCount;
private Integer updateMaxDocCount; //ASB ... 4 012 Separate update document process
private Integer importMaxRunTimeMinutes;
private String importAuditFileFolder;
private Integer updatingBatchSize;
// ASB ... 2 022 Replicate Parameters
//private String readOnlyGroup;
private String excludedGroup;
private String excludedUser;
private String startSearchDate;
private String deltaSearchDate; //ASB ... 4 009
private String deltaHours = "01"; //ASB ... 4 020
private Integer maxSearchSize;
//ASB ... 4 020 Debug Flag for performance improvement
private String debugFlag;
//ASB ... 4 027 Get LDAP Search Flag for performance improvement
private String LDAPSearchFlag;
//ASB ... 4 022 Get Folder Subclass list
private String folderSubclasses;
private XPath xPath;
public static void main(String args[]) throws Exception {
CEReplicateConfig cemc = new CEReplicateConfig();
}
public CEReplicateConfig() throws Exception {
XPathFactory factory= XPathFactory.newInstance();
xPath=factory.newXPath();
xPath.reset();
getDocument();
readConfig();
}
private void readConfig() throws Exception {
//ASB ... 2 022 Replication Configuration Parameters
//readOnlyGroup = getXMLVal("/config/cereplicate/readonlygroup/text()");
excludedGroup = getXMLVal("/config/cereplicate/excludedgroup/text()");
excludedUser = getXMLVal("/config/cereplicate/excludeduser/text()");
startSearchDate = getXMLVal("/config/cereplicate/startsearchdate/text()");
maxSearchSize = Integer.parseInt(getXMLVal("/config/cereplicate/MaxSearchSize/text()"));
//ASB ... 4 020 Retrieve Debug Output flag (off/on) value
debugFlag = getXMLVal("/config/cereplicate/DebugOutputFlag/text()");
//ASB ... 4 27 Retrieve LDAP Search flag (off/on) value
LDAPSearchFlag = getXMLVal("/config/cereplicate/LDAPSearchFlag/text()");
deltaHours = getXMLVal("/config/cereplicate/DeltaHours/text()");
//ASB ... 4 022 Add FolderSubclasses retrieval
folderSubclasses = getXMLVal("/config/cereplicate/FolderSubclasses/text()");
// Export Parameters
exportOSName = getXMLVal("/config/ceexport/osname/text()");
exportCEUser = getXMLVal("/config/ceexport/ceuser/text()");
exportCEPassword = getXMLVal("/config/ceexport/cepassword/text()");
//ASB ... 4 001 Not in the form Encrypt3dpasswordvalueP4ssw0rd
String decryptedExportCEPassword = decrypt(exportCEPassword);
if (decryptedExportCEPassword.startsWith("Encrypt3d")&& decryptedExportCEPassword.endsWith("P4ssw0rd") ){
//ASB ... 4 001 Extract the actual password
//
exportCEPassword = decryptedExportCEPassword.substring(9, decryptedExportCEPassword.length()-8);
}else{
//ASB ... 4 001 update the clear password
String encryptedExportCEPassword = "Encrypt3d" + exportCEPassword + "P4ssw0rd";
encryptedExportCEPassword = encrypt(encryptedExportCEPassword);
System.out.print("
ENCRYPTED CE Export PASSWORD = " + encryptedExportCEPassword);
System.out.print("
");
// ASB ... 4 001 Write encrypted password back to the config.xml file
// ASB Note Forward slashes for a Linux path!!
String file = System.getProperty("user.dir") + "/config/config.xml"; //String file = "config.xml";
// SET file Name, root element, tag element, old value, new value
updateXML(file,"ceexport","cepassword", exportCEPassword, encryptedExportCEPassword);
}
exportMaxDocCount = Integer.parseInt(getXMLVal("/config/ceexport/MaxDocCount/text()"));
exportMaxRunTimeMinutes = Integer.parseInt(getXMLVal("/config/ceexport/MaxRunTimeMinutes/text()"));
exportMaxErrorCount = Integer.parseInt(getXMLVal("/config/ceexport/MaxConsecutiveErrors/text()"));
exportAuditFile = getXMLVal("/config/ceexport/AuditFile/text()");
String tmp = getXMLVal("/config/ceimport/UpdatingBatchSize/text()");
if (tmp.length() > 0){
try {
updatingBatchSize = Integer.parseInt(tmp);
}
catch (Exception e){
updatingBatchSize = 250;
}
}
else {
updatingBatchSize = 250;
}
// import parameters
importOSName = getXMLVal("/config/ceimport/osname/text()");
importOSRootFolder = getXMLVal("/config/ceimport/osrootfolder/text()");
importCEUrl = getXMLVal("/config/ceimport/ceurl/text()");
importCEStanza = getXMLVal("/config/ceimport/cestanza/text()");
importCELoginConfig = getXMLVal("/config/ceimport/celoginconfig/text()");
importCEUser = getXMLVal("/config/ceimport/ceuser/text()");
importCEPassword = getXMLVal("/config/ceimport/cepassword/text()");
//ASB ... 4 001 Check if this is still clear ie
//ASB ... 4 001 Not in the form Encrypt3dpasswordvalueP4ssw0rd
String decryptedImportCEPassword = decrypt(importCEPassword);
if (decryptedImportCEPassword.startsWith("Encrypt3d")&& decryptedImportCEPassword.endsWith("P4ssw0rd") ){
//ASB ... 4 001 Extract the actual password
//
importCEPassword = decryptedImportCEPassword.substring(9, decryptedImportCEPassword.length()-8);
}else{
//ASB ... 4 001 update the clear password
String encryptedImportCEPassword = "Encrypt3d" + importCEPassword + "P4ssw0rd";
encryptedImportCEPassword = encrypt(encryptedImportCEPassword);
System.out.print("
ENCRYPTED CE IMPORT PASSWORD = " + encryptedImportCEPassword);
System.out.print("
");
// ASB ... 4 001 Write encrypted password back to the config.xml file
String file = System.getProperty("user.dir") + "/config/config.xml";
// SET file Name, root element, tag element, old value, new value
updateXML(file,"ceimport","cepassword", importCEPassword, encryptedImportCEPassword);
}
importMaxErrorCount = Integer.parseInt(getXMLVal("/config/ceimport/MaxConsecutiveErrors/text()"));
importMaxDocCount = Integer.parseInt(getXMLVal("/config/ceimport/MaxDocCount/text()"));
importMaxRunTimeMinutes = Integer.parseInt(getXMLVal("/config/ceimport/MaxRunTimeMinutes/text()"));
importAuditFile = getXMLVal("/config/ceimport/AuditFile/text()");
importAuditFileFolder= getXMLVal("/config/ceimport/AuditFileFolders/text()");
//ASB ... 3 003 Get Delete Document and Delete Folder parameters
deleteMaxDocCount = Integer.parseInt(getXMLVal("/config/cereplicate/deleteMaxDocCount/text()"));
deleteDocumentAuditFile = getXMLVal("/config/cereplicate/DeleteAuditFile/text()");
deleteMaxErrorCount = Integer.parseInt(getXMLVal("/config/cereplicate/MaxConsecutiveDeleteErrors/text()"));
deleteMaxFolderCount = Integer.parseInt(getXMLVal("/config/cereplicate/DeleteMaxFolderCount/text()"));
deleteFolderAuditFile = getXMLVal("/config/cereplicate/DeleteFolderAuditFile/text()");
//ASB ... 4 012 Get Update Document parameters
updateMaxDocCount = Integer.parseInt(getXMLVal("/config/cereplicate/UpdateMaxDocCount/text()"));
updateDocumentAuditFile = getXMLVal("/config/cereplicate/UpdateDocumentAuditFile/text()");
updateMaxErrorCount = Integer.parseInt(getXMLVal("/config/cereplicate/MaxConsecutiveUpdateErrors/text()"));
}
public void updateProcessDate(Date processDate){
// ASB ... 4 001 Write Date String back to the config.xml file
// Need date in the format 20130923T125628Z
String file = System.getProperty("user.dir") + "/config/config.xml";
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
//Need 24 Hour clock 00 -> 23 !! above is 12 hour
SimpleDateFormat sdfTime = new SimpleDateFormat("HHmmss");
String newStartSearchDate = sdf.format(processDate)+ "T" + sdfTime.format(processDate)+ "Z";
updateXML(file,"cereplicate","startsearchdate", startSearchDate, newStartSearchDate);
}
private void getDocument() throws Exception {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = null;
try {
db = dbf.newDocumentBuilder();
} catch (ParserConfigurationException e) {
logger.error(e);
throw e;
}
//ASB ... 4 001 Normalise path to the config we want to use
InputStream in = new FileInputStream(System.getProperty("user.dir") + "/config/config.xml");
dom = db.parse(in);
in.close(); //ASB ... 4 001 Now close the config.xml stream
}
public String getXMLVal(String expression) throws Exception {
Node n = (Node) xPath.evaluate(expression, dom, XPathConstants.NODE);
if (n != null)
return n.getNodeValue();
return "";
}
//ASB ... 4 001 Put new value into the dom node
private void updateXML(String file, String mainElement, String sTag, String strOldValue, String strNewValue) {
//DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
Document doc = null;
try {
DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
doc = docBuilder.parse(file);
// Get the root element
Node config = doc.getFirstChild();
// Get the main element by tag name directly
Node ceMain = doc.getElementsByTagName(mainElement).item(0);
// loop the configuration main section child node
NodeList list = ceMain.getChildNodes();
for (int i = 0; i < list.getLength(); i++) {
Node node = list.item(i);
// get the passaword element, and update the value
if (sTag.equals(node.getNodeName())) {
node.setTextContent(strNewValue);
}
}
} catch (ParserConfigurationException pce) {
pce.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (SAXException sae) {
sae.printStackTrace();
}
catch(Exception ex) {
String exError = ex.getMessage();
}
File filer = new File(file);
saveToXML(filer,doc);
} // Save the updated DOM into the XML back
// Save the updated DOM into the XML back
private void saveToXML(File file, Document doc) {
try {
TransformerFactory factory = TransformerFactory.newInstance();
Transformer transformer = factory.newTransformer();
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
StringWriter writer = new StringWriter();
StreamResult result = new StreamResult(writer);
DOMSource source = new DOMSource(doc);
transformer.transform(source, result);
String strTemp = writer.toString();
FileWriter fileWriter = new FileWriter(file);
BufferedWriter bufferedWriter = new BufferedWriter(fileWriter);
bufferedWriter.write(strTemp);
bufferedWriter.flush();
bufferedWriter.close();
}
catch(Exception ex) {
}
}
public String[] getXMLVals(String expression) throws Exception {
Vector vtmp = null;
NodeList nl = null;
vtmp = new Vector();
nl = (NodeList) xPath.evaluate(expression, dom, XPathConstants.NODESET);
for (int x = 0; x < nl.getLength(); x ++){
vtmp.add(nl.item(x).getNodeValue());
}
return (String[])vtmp.toArray(new String[vtmp.size()]);
}
public String getExportOSName(){
return exportOSName;
}
public String getExportCEUser(){
return exportCEUser;
}
public String getExportCEPassword(){
return exportCEPassword;
}
public Integer getExportMaxDocCount(){
return exportMaxDocCount;
}
//ASB ... 3 003 Add Getter for exportMaxFolderCount the maximum number of Folders to be deleted
public Integer getDeleteMaxFolderCount(){
return deleteMaxFolderCount;
}
public Integer getExportMaxRunTimeMinutes(){
return exportMaxRunTimeMinutes;
}
public Integer getMaxSearchSize(){
return maxSearchSize; //ASB ... 2 022
}
public String getExcludedGroup(){
return excludedGroup; //ASB ... 4 007
}
public String getExcludedUser(){
return excludedUser; //ASB ... 4 007
}
public String getDebugOutputFlag(){
return debugFlag; //ASB ... 4 020
}
public String getLDAPSearchFlag(){
return LDAPSearchFlag; //ASB ... 4 027
}
public String getDeltaHours(){
return deltaHours; //ASB ... 4 020
}
public String getFolderSubclasses(){
return folderSubclasses; //ASB ... 4 007
}
public String getStartSearchDate(){
Date processDate = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd",Locale.ENGLISH); //ASB ... 4 009 Added Locale
Date javaStartDate = null;
//Need 24 Hour clock 00 -> 23 !! hh is 12 hour
SimpleDateFormat sdfTime = new SimpleDateFormat("HHmmss",Locale.ENGLISH); //ASB ... 4 009 Added Locale
//deltaSearchDate = sdf.format(processDate)+ "T" + sdfTime.format(processDate).substring(0,2)+"0000Z";
//ASB ... 4 013 Temporary change to midnight delta
//deltaSearchDate = sdf.format(processDate)+ "T" + sdfTime.format(processDate)+ "Z";
//deltaSearchDate = sdf.format(processDate)+ "T000000Z";
//ASB ... 4 009 Check if the current date was manually set - ie is greater
// than one hour difference from the current time.
// If date was set manually then use this date instead as the search date.
DateFormat diffDateFormat = DateFormat.getDateInstance();
String sDiffDate = startSearchDate.substring(0, 8);
String sDiffTime = startSearchDate.substring(9,15);
Date diffDate = null;
Date diffTime = null;
Date dDeltaTime = null;
Long dateSeconds = null;
try {
//ASB ... 4 010 Need to subtract off the java start date
javaStartDate = sdf.parse("19700101"); //ASB ... 4 010
diffDate = sdf.parse(sDiffDate) ;
diffTime = sdfTime.parse(sDiffTime) ;
//Test for zero
Long searchhours = (long) 0;
if((diffTime.getTime() - javaStartDate.getTime() - Long.parseLong(deltaHours)*3600000) > 0)
searchhours =(diffTime.getTime() - javaStartDate.getTime() - Long.parseLong(deltaHours)*3600000)/3600000;
String sSearchHours = searchhours.toString();
if (sSearchHours.length() == 1)sSearchHours = "0"+ sSearchHours;
dDeltaTime = sdfTime.parse(sSearchHours + "0000");
deltaSearchDate = sdf.format(processDate) + "T" + sSearchHours + "0000Z";
//ASB ... 4 010 Added subtraction of the Java Start Date milliseconds
//dateSeconds = diffDate.getTime() + diffTime.getTime() - javaStartDate.getTime();
dateSeconds = (diffDate.getTime() + (diffTime.getTime()) - (dDeltaTime.getTime()));
} catch (ParseException e) {
// Should be a controlled format - set to default if we get here
e.printStackTrace();
}
//ASB ... 4 009 Get Process Date in milliseconds
Long milliSecondsLastDate = processDate.getTime();
//ASB ... 4 013 Temporary change to midnight delta //WAS 3600000
//if ((milliSecondsLastDate - dateSeconds) < 86400000 ){
//ASB ... 4 020 Allow 30 Minute delta on top
if ((milliSecondsLastDate - dateSeconds) < (diffTime.getTime()- javaStartDate.getTime() )){
startSearchDate = deltaSearchDate; //Allow 60 Minutes to fix missing documents
}
return startSearchDate; //ASB ... 2 022
}
public Integer getExportMaxErrorCount(){
return exportMaxErrorCount;
}
//ASB ... 3 003 Retrieve the maximum allowed Delete Document errors
public Integer getDeleteMaxErrorCount(){
return deleteMaxErrorCount;
}
//ASB ... 4 012 Retrieve the maximum allowed Update Document errors
public Integer getUpdateMaxErrorCount(){
return updateMaxErrorCount;
}
public String getExportAuditFile(){
return exportAuditFile;
}
public String getImportOSName(){
return importOSName;
}
public String getImportOSRootFoler(){
return importOSRootFolder;
}
public String getImportCEUrl(){
return importCEUrl;
}
public String getImportCEStanza(){
return importCEStanza;
}
public String getImportCELoginConfig(){
return importCELoginConfig;
}
public String getImportCEUser(){
return importCEUser;
}
public String getImportCEPassword(){
return importCEPassword;
}
public Integer getImportMaxErrorCount(){
return importMaxErrorCount;
}
public Integer getImportMaxDocCount(){
return importMaxDocCount;
}
//ASB ... 3 003 Getter Method for Delete Max Document Count
public Integer getDeleteMaxDocCount(){
return deleteMaxDocCount;
}
//ASB ... 4 012 Getter Method for Update Max Document Count
public Integer getUpdateMaxDocCount(){
return updateMaxDocCount;
}
public Integer getImportMaxRunTimeMinutes(){
return importMaxRunTimeMinutes;
}
public String getImportAuditFile(){
return importAuditFile;
}
//ASB ... 3 003 Set Delete Document Audit File
public String getDeleteDocumentAuditFile(){
return deleteDocumentAuditFile;
}
//ASB ... 3 003 Set Delete Folder Audit File
public String getDeleteFolderAuditFile(){
return deleteFolderAuditFile;
}
//ASB ... 4 012 Set Update Document Audit File
public String getUpdateDocumentAuditFile(){
return updateDocumentAuditFile;
}
public String getImportAuditFileFolders(){
return importAuditFileFolder;
}
public int getUpdatingBatchSize(){
return updatingBatchSize;
}
private static String encrypt(String str) {
try {
//encoding byte array into base 64
byte[] encoded = Base64.encodeBase64(str.getBytes());
String encString = new String(encoded);
return encString;
} catch (Exception e) {
}
return null;
}
private static String decrypt(String str) {
try {
//decoding byte array into base64
byte[] decoded = Base64.decodeBase64(str);
String sDecoded = new String(decoded);
return sDecoded;
} catch (Exception e) {
}
return null;
}
}
Part 5 – Code Listing – CEMigrateConfig
The CEMigrateConfig.java Java code file supports the Main Java Replication program by retrieving the Object Store administration username and password credentials. On a first call to access the config.xml file, it checks if the password requires encryption and updates the config.xml to store an encrypted version of the password for additional security.
It also reads in the replication XML parameters in the config.xml file for the search date and source and target object store names and the root folder name to be replicated.
CEMigrateConfig – 425 Lines of Java Code – fn_common.jar
package com.ibm.ce.utils;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.Vector;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;
//ASB ... 4 001 Encryption / Decryption Java Classes
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import java.io.UnsupportedEncodingException;
import java.security.spec.AlgorithmParameterSpec;
import java.security.spec.KeySpec;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;
import javax.lang.model.element.Element;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.StringWriter;
import java.net.URL;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
//ASB ... 4 END
import org.apache.commons.codec.binary.Base64;
import org.apache.log4j.Logger;
import org.apache.xml.serialize.XMLSerializer;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
// ASB Software Development Limited - import updated to point to the Apache equivalent class 29/07/2022
//import com.sun.org.apache.xml.internal.serialize.OutputFormat;
import org.apache.xml.serialize.OutputFormat;
public class CEMigrateConfig {
Logger logger = Logger.getLogger(CEMigrateConfig.class.getName());
private Document dom;
private String sourceCEDBDriverClasses[];
private String sourceCEDBUserName;
private String sourceCEDBPassword;
private String sourceCEDBURL;
//ASB ... 3 001 - if needed!
private String targetCEDBURL;
private String targetCEDBUserName;
private String targetCEDBPassword;
private String targetStanza;
private String CEMigrateDBDriverClasses[];
private String CEMigrateDBUserName;
private String CEMigrateDBPassword;
private String CEMigrateDBURL;
// Export Parameters
private boolean isOracle;
private String exportPath;
private String exportOSName;
private String exportCEUser;
private String exportCEPassword;
private String exportSql;
private Integer exportMaxDocCount;
private Integer exportMaxRunTimeMinutes;
private Integer exportMaxErrorCount;
private String exportSelectFromDocsForExportSQL;
private String exportAuditFile;
private Integer exportFeedBatchSize;
private int bytesize;
private int threadCount;
private int threadPoolMaxQueueSize;
private int SQLBatchSize;
//ASB ... 3 001 Export parameters
private String exportCEUrl;
private String exportCELoginConfig; //ASB
private String exportCEWaspLocation;//ASB
private String exportContainmentNameExtension; //ASB 01/03/2011 Switch = Yes then set Containment Name Extension
private String exportCEStanza;
// Import parameters
private String importPath;
private String importOSName;
private String importOSRootFolder;
private String importCEUrl;
private String importCELoginConfig; //ASB
private String importCEWaspLocation;//ASB
private String importContainmentNameExtension; //ASB 01/03/2011 Switch = Yes then set Containment Name Extension
private String importCEStanza;
private String importCEUser;
private String importCEPassword;
private Integer importMaxErrorCount;
private String importAuditFile;
private String importSelectFromDocsForImportSQL;
private Integer importMaxDocCount;
private Integer importMaxRunTimeMinutes;
private String importAuditFileFolder;
private Integer updatingBatchSize;
private XPath xPath;
public static void main(String args[]) throws Exception {
CEMigrateConfig cemc = new CEMigrateConfig();
}
public CEMigrateConfig() throws Exception {
XPathFactory factory= XPathFactory.newInstance();
xPath=factory.newXPath();
xPath.reset();
getDocument();
readConfig();
}
private void readConfig() throws Exception {
// Export Parameters
//exportPath = getXMLVal("/config/ceexport/exportpath/text()");
exportOSName = getXMLVal("/config/ceexport/osname/text()");
exportCEUser = getXMLVal("/config/ceexport/ceuser/text()");
//ASB ... 4 001 Password is returned encrypted now
exportCEPassword = getXMLVal("/config/ceexport/cepassword/text()");
//ASB ... 4 001 Not in the form Encrypt3dpasswordvalueP4ssw0rd
String decryptedExportCEPassword = decrypt(exportCEPassword);
if (decryptedExportCEPassword.startsWith("Encrypt3d")&& decryptedExportCEPassword.endsWith("P4ssw0rd") ){
//ASB ... 4 001 Extract the actual password
//
exportCEPassword = decryptedExportCEPassword.substring(9, decryptedExportCEPassword.length()-8);
}else{
//ASB ... 4 001 update the clear password
String encryptedExportCEPassword = "Encrypt3d" + exportCEPassword + "P4ssw0rd";
encryptedExportCEPassword = encrypt(encryptedExportCEPassword);
System.out.print("
ENCRYPTED CE Export PASSWORD = " + encryptedExportCEPassword);
System.out.print("
");
// ASB ... 4 001 Write encrypted password back to the config.xml file
String file = System.getProperty("user.dir") + "/config/config.xml";
//String file = "config.xml";
// SET file Name, root element, tag element, old value, new value
updateXML(file,"ceexport","cepassword", exportCEPassword, encryptedExportCEPassword);
}
exportMaxDocCount = Integer.parseInt(getXMLVal("/config/ceexport/MaxDocCount/text()"));
exportMaxRunTimeMinutes = Integer.parseInt(getXMLVal("/config/ceexport/MaxRunTimeMinutes/text()"));
exportMaxErrorCount = Integer.parseInt(getXMLVal("/config/ceexport/MaxConsecutiveErrors/text()"));
exportAuditFile = getXMLVal("/config/ceexport/AuditFile/text()");
String tmp = getXMLVal("/config/ceimport/UpdatingBatchSize/text()");
if (tmp.length() > 0){
try {
updatingBatchSize = Integer.parseInt(tmp);
}
catch (Exception e){
updatingBatchSize = 250;
}
}
else {
updatingBatchSize = 250;
}
// import parameters
importOSName = getXMLVal("/config/ceimport/osname/text()");
importOSRootFolder = getXMLVal("/config/ceimport/osrootfolder/text()");
importCEUrl = getXMLVal("/config/ceimport/ceurl/text()");
importCEStanza = getXMLVal("/config/ceimport/cestanza/text()");
importCELoginConfig = getXMLVal("/config/ceimport/celoginconfig/text()");
importCEUser = getXMLVal("/config/ceimport/ceuser/text()");
//ASB ... 4 001 Password is returned encrypted now
importCEPassword = getXMLVal("/config/ceimport/cepassword/text()");
//ASB ... 4 001 Check if this is still clear ie
//ASB ... 4 001 Not in the form Encrypt3dpasswordvalueP4ssw0rd
String decryptedImportCEPassword = decrypt(importCEPassword);
if (decryptedImportCEPassword.startsWith("Encrypt3d")&& decryptedImportCEPassword.endsWith("P4ssw0rd") ){
//ASB ... 4 001 Extract the actual password
//
importCEPassword = decryptedImportCEPassword.substring(9, decryptedImportCEPassword.length()-8);
}else{
//ASB ... 4 001 update the clear password
String encryptedImportCEPassword = "Encrypt3d" + importCEPassword + "P4ssw0rd";
encryptedImportCEPassword = encrypt(encryptedImportCEPassword);
System.out.print("
ENCRYPTED CE IMPORT PASSWORD = " + encryptedImportCEPassword);
System.out.print("
");
// ASB ... 4 001 Write encrypted password back to the config.xml file
String file = System.getProperty("user.dir") + "/config/config.xml";
// SET file Name, root element, tag element, old value, new value
updateXML(file,"ceimport","cepassword", importCEPassword, encryptedImportCEPassword);
}
importMaxErrorCount = Integer.parseInt(getXMLVal("/config/ceimport/MaxConsecutiveErrors/text()"));
importMaxDocCount = Integer.parseInt(getXMLVal("/config/ceimport/MaxDocCount/text()"));
importMaxRunTimeMinutes = Integer.parseInt(getXMLVal("/config/ceimport/MaxRunTimeMinutes/text()"));
importAuditFile = getXMLVal("/config/ceimport/AuditFile/text()");
importAuditFileFolder= getXMLVal("/config/ceimport/AuditFileFolders/text()");
//ASB ... 3 001 export parameters for Session Object
exportCEUrl = getXMLVal("/config/ceexport/ceurl/text()");
exportCEStanza = getXMLVal("/config/ceexport/cestanza/text()"); //null
exportCELoginConfig = getXMLVal("/config/ceexport/celoginconfig/text()");
}
private void getDocument() throws Exception {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = null;
try {
db = dbf.newDocumentBuilder();
} catch (ParserConfigurationException e) {
logger.error(e);
throw e;
}
//ASB ... 4 001 Normalise path to the config we want to use
InputStream in = new FileInputStream(System.getProperty("user.dir") + "/config/config.xml");
dom = db.parse(in);
in.close(); //ASB ... 4 001 Now close the confog.xml file
}
//ASB ... 4 001 Create new Write method
// Save the updated DOM into the XML back
public String getXMLVal(String expression) throws Exception {
Node n = (Node) xPath.evaluate(expression, dom, XPathConstants.NODE);
if (n != null)
return n.getNodeValue();
return "";
}
//ASB ... 4 001 Put new value into the dom node
private void updateXML(String file, String mainElement, String sTag, String strOldValue, String strNewValue) {
Document doc = null;
try {
DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
doc = docBuilder.parse(file);
// Get the root element
Node config = doc.getFirstChild();
// Get the main element by tag name directly
Node ceMain = doc.getElementsByTagName(mainElement).item(0);
// loop the configuration main section child node
NodeList list = ceMain.getChildNodes();
for (int i = 0; i < list.getLength(); i++) {
Node node = list.item(i);
// get the passaword element, and update the value
if (sTag.equals(node.getNodeName())) {
node.setTextContent(strNewValue);
}
}
} catch (ParserConfigurationException pce) {
pce.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (SAXException sae) {
sae.printStackTrace();
}
catch(Exception ex) {
String exError = ex.getMessage();
}
File filer = new File(file);
saveToXML(filer,doc);
} // Save the updated DOM into the XML back
// Save the updated DOM into the XML back
private void saveToXML(File file, Document doc) {
try {
TransformerFactory factory = TransformerFactory.newInstance();
Transformer transformer = factory.newTransformer();
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
StringWriter writer = new StringWriter();
StreamResult result = new StreamResult(writer);
DOMSource source = new DOMSource(doc);
transformer.transform(source, result);
String strTemp = writer.toString();
FileWriter fileWriter = new FileWriter(file);
BufferedWriter bufferedWriter = new BufferedWriter(fileWriter);
bufferedWriter.write(strTemp);
bufferedWriter.flush();
bufferedWriter.close();
}
catch(Exception ex) {
}
}
public String[] getXMLVals(String expression) throws Exception {
Vector vtmp = null;
NodeList nl = null;
vtmp = new Vector();
nl = (NodeList) xPath.evaluate(expression, dom, XPathConstants.NODESET);
for (int x = 0; x < nl.getLength(); x ++){
vtmp.add(nl.item(x).getNodeValue());
}
return (String[])vtmp.toArray(new String[vtmp.size()]);
}
public String getExportOSName(){
return exportOSName;
}
public String getExportCEUser(){
return exportCEUser;
}
public String getExportCEPassword(){
return exportCEPassword;
}
public Integer getExportMaxDocCount(){
return exportMaxDocCount;
}
public Integer getExportMaxRunTimeMinutes(){
return exportMaxRunTimeMinutes;
}
public Integer getExportMaxErrorCount(){
return exportMaxErrorCount;
}
public String getExportAuditFile(){
return exportAuditFile;
}
public String getImportOSName(){
return importOSName;
}
public String getImportOSRootFoler(){
return importOSRootFolder;
}
//ASB ... 3 001 add getter Methods for export Session Object
public String getExportCEUrl(){
return exportCEUrl;
}
public String getExportCEStanza(){
return exportCEStanza;
}
public String getExportCELoginConfig(){
return exportCELoginConfig;
}
//ASB ... 3 001 add END
public String getImportCEUrl(){
return importCEUrl;
}
public String getImportCEStanza(){
return importCEStanza;
}
public String getImportCELoginConfig(){
return importCELoginConfig;
}
public String getImportCEUser(){
return importCEUser;
}
public String getImportCEPassword(){
return importCEPassword;
}
public Integer getImportMaxErrorCount(){
return importMaxErrorCount;
}
public Integer getImportMaxDocCount(){
return importMaxDocCount;
}
public Integer getImportMaxRunTimeMinutes(){
return importMaxRunTimeMinutes;
}
public String getImportAuditFile(){
return importAuditFile;
}
public String getImportAuditFileFolders(){
return importAuditFileFolder;
}
public int getUpdatingBatchSize(){
return updatingBatchSize;
}
private static String encrypt(String str) {
try {
//encoding byte array into base 64
byte[] encoded = Base64.encodeBase64(str.getBytes());
String encString = new String(encoded);
return encString;
} catch (Exception e) {
}
return null;
}
private static String decrypt(String str) {
try {
//decoding byte array into base64
byte[] decoded = Base64.decodeBase64(str);
String sDecoded = new String(decoded);
return sDecoded;
} catch (Exception e) {
}
return null;
}
}
Part 6 – Code Listing – Supporting Utility Code
This PropsUtil Java code is one of a number of general supporting utility methods described in this part. The code is used to retrieve properties from a cebase.properties file containing external parameters to support access to Object Stores. (It is not used in the example replication program described in this chapter, but is bundled as a utility method.)
PropsUtil – 68 Lines of Java Code – fn_utils.jar
package com.asb.utils;
import org.apache.log4j.Logger;
import java.util.Properties;
import java.io.IOException;
import java.io.InputStream;
import java.io.FileInputStream;
/**
*
*/
public class PropsUtil {
public PropsUtil() {}
public String getProperty(Properties theProperties, String strProperty, String strDefault){
if (theProperties.getProperty(strProperty) != null){
return theProperties.getProperty(strProperty);
}
else {
theLog.warn("Did not find " + strProperty + " property in cebase.properties");
theProperties.setProperty(strProperty, strDefault);
return strDefault;
}
}
public Properties getProps(String propFile) throws Exception {
return pgetProps(propFile,false);
}
public Properties getProps(Properties theProps, String propFile) throws Exception {
return pgetProps(theProps,propFile,false);
}
public Properties getProps(String propFile, boolean fullPath) throws Exception {
return pgetProps(propFile,fullPath);
}
private Properties pgetProps(String propFile, boolean fullPath) throws Exception {
Properties theProps = new Properties();
return pgetProps(theProps, propFile, fullPath);
}
private Properties pgetProps(Properties theProps, String propFile, boolean fullPath) throws Exception {
if (theProps == null)
theProps = new Properties();
InputStream in = null;
if (fullPath == false){
in = PropsUtil.class.getClassLoader().getResourceAsStream(propFile);
}
else {
in = new FileInputStream(propFile);
}
try {
theProps.load(in);
}
catch (IOException e) {
theLog.debug(e.getMessage());
}
catch (NullPointerException e){
Exception theE = new Exception("Properties file " + propFile + " not found");
throw theE;
}
return theProps;
}
static Logger theLog = Logger.getLogger(PropsUtil.class.getName());
}
The CEConnect Java code is one of a number of general supporting utility methods described in this part. The method is a helper used to facilitate connection to the IBM FileNet Object stores through the Web Service Interface SOAP calls.
CEConnect – 67 Lines of Java Code – fn_connect.jar
package com.asb.filenet.ce;
import javax.security.auth.Subject;
//import org.apache.log4j.Logger;
import com.filenet.api.core.Connection;
import com.filenet.api.core.Factory;
import com.filenet.api.util.UserContext;
public class CEConnect {
private String stanza = null;
private Connection conn = null;
private UserContext uc = null;
private String ceUri = null;
private Subject subject = null;
//private static Logger logger = Logger.getLogger(CEConnect.class.getName());
/**
*
* @param ceUrl
*/
/**
* @param ceUrl
*/
public CEConnect(String ceUrl){
this.stanza = "FileNetP8WSI";
this.ceUri = ceUrl;
connect();
}
/**
* @param ceUrl
* @param stanza
*/
public CEConnect(String ceUrl, String stanza){
this.stanza = stanza;
this.ceUri = ceUrl;
connect();
}
private void connect(){
conn = Factory.Connection.getConnection(this.ceUri);
uc = UserContext.get();
}
/**
* @param userName
* @param password
*/
public void logon(String userName, String password){
subject = UserContext.createSubject(conn, userName, password, this.stanza);
uc.pushSubject(subject);
}
/**
* @return
*/
public String getCEUri(){
return this.ceUri;
}
public Connection getConnection(){
return conn;
}
public void popSubject(){
uc.popSubject();
}
public void pushSubject(){
uc.pushSubject(subject);
}
}
CEConnection – 119 Lines of Java Code – fn_connection.jar
The CEConnection Java code is one of a number of general supporting utility methods described in this part. It is the main method called by the Replication program to sign in the IBM FileNet administration user to the source and target Object Stores. It also sets the necessary user security access to give special access levels to enable the Document and Folder object system properties for Date Created and Date Modified and the unique GUID identifier to be overwritten to create an exact replica of the source objects.
package com.ibm.filenet.ce.CEConnection;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import com.asb.filenet.ce.CEConnect;
import com.filenet.api.collection.AccessPermissionList;
import com.filenet.api.constants.AccessRight;
import com.filenet.api.constants.AccessType;
import com.filenet.api.constants.RefreshMode;
import com.filenet.api.core.Domain;
import com.filenet.api.core.Factory;
import com.filenet.api.core.Factory.AccessPermission;
import com.filenet.api.core.ObjectStore;
import com.ibm.ce.utils.CEMigrateConfig;
//import com.ibm.ce.utils.Config;
//ASB Software Development Limited 30th July 2022 - Added as below
import com.ibm.ce.utils.CEReplicateConfig;
public class CEConnection {
private static String uri;
private static String username;
private static String password;
private static String stanza;
private static String targetURI;
private static String targetUserName;
private static String targetPassword;
private static String targetStanza;
private CEConnect ceConnect = null;
private Domain d = null;
private static Logger logger = Logger.getLogger(CEConnection.class.getName());
static {
try {
CEMigrateConfig cemc = new CEMigrateConfig();
uri = cemc.getExportCEUrl();
username = cemc.getExportCEUser();//Administrator
password = cemc.getExportCEPassword(); //filenet
stanza = cemc.getExportCEStanza(); //null
targetURI = cemc.getImportCEUrl();
targetUserName = cemc.getImportCEUser();
targetPassword = cemc.getImportCEPassword(); //filenet
targetStanza = cemc.getImportCEStanza();
//ASB Need these for session
System.setProperty("java.security.auth.login.config",cemc.getImportCELoginConfig());
}
catch (Exception e){
logger.error(e.getMessage(),e);
}
}
public CEConnection(String sourceType){
//ASB ... 3 Check if this is Source or Target Object Store
if(sourceType.equalsIgnoreCase("Source")){
ceConnect = new CEConnect(uri, stanza);
ceConnect.logon(username, password);
d = Factory.Domain.fetchInstance(ceConnect.getConnection(),"",null);
}else{ //Target Object Store - get Target parameters
ceConnect = new CEConnect(targetURI, targetStanza);
ceConnect.logon(targetUserName, targetPassword);
d = Factory.Domain.fetchInstance(ceConnect.getConnection(),"",null);
}
}
public void popSubject(){
ceConnect.popSubject();
}
public void pushSubect(){
ceConnect.pushSubject();
}
public ObjectStore getObjectStore(String osName){
ObjectStore os = Factory.ObjectStore.fetchInstance(d,osName, null);
//ObjectStore os = (ObjectStore)oStores.get(osName);
logger.log(Level.DEBUG, "Got ObjectStore " + os.get_Name());
// Add special rights to the Object Store
setAccessRights(
os.get_Domain(),
username, // Example: "CEMPAdmin"
osName); // Example: "ObjectStore1"
return os;
}
public Domain getDomain(){
return d;
}
@SuppressWarnings("static-access")
public static void setAccessRights(
Domain domain,
String granteeName, // Example: "CEMPAdmin"
String objStoreName) // Example: "ObjectStore1"
{
final int ACCESS_REQUIRED = AccessRight.WRITE_ANY_OWNER.getValue() |
AccessRight.REMOVE_OBJECTS.getValue() | AccessRight.STORE_OBJECTS.getValue() |
AccessRight.CONNECT.getValue() | AccessRight.WRITE_ACL.getValue() |
AccessRight.READ_ACL.getValue() | AccessRight.MODIFY_OBJECTS.getValue() | AccessRight.PRIVILEGED_WRITE_AS_INT;
// Create a new access permission object.
com.filenet.api.security.AccessPermission ap = Factory.AccessPermission.createInstance();
// Set access permissions.
ap.set_GranteeName(granteeName);
ap.set_AccessType(AccessType.ALLOW);
ap.set_AccessMask(new Integer(ACCESS_REQUIRED));
// Set and save the new permissions.
ObjectStore objStore = Factory.ObjectStore.fetchInstance(domain, objStoreName, null);
AccessPermissionList apl = objStore.get_Permissions();
apl.add(ap);
objStore.set_Permissions(apl);
objStore.save(RefreshMode.REFRESH);
}
}
Chapter 4 Exercises
The following questions relate to the Replication Java program, CEReplicate class, and its supporting Java library components we covered in this chapter.
Multiple Choice Questions
- 1.What shell script would you run to copy the documents and folders from a source Object Store to a Target replicated Object Store?
- a)
envCommonReplicate.sh
- b)
Replicate.sh
- c)
http://ecmukdemo6:9080/acce/
- d)
env5x.sh
- 2.Which file is used to provide the Java for XML parsing requirements for the Java replication program?
- a)
config.xml
- b)
Jace.jar
- c)
xerces.jar
- d)
fn_utils.jar
- 3.In which folder can the Replication.sh file be found?
- a)
/opt/replication/config
- b)
/opt/replication
- c)
/opt/replication/libs
- d)
/opt/replication/jars
- 4.Which of the following Events are not required for replicating the Folder Objects?
- a)
Update Security Event
- b)
Unfile Event
- c)
Checkin Event
- d)
Deletion Event
Multiple Choice Answers
- 1.
b) Replicate.sh
- 2.
c) xerces.jar
- 3.
b) /opt/replication
- 4.
c) Checkin Event
Questions
- 1.
Outline what steps you would use to validate that the Replication program was functioning correctly. Describe the tests you would perform to prove the correct functionality for repeat runs of the Replication program.
- 2.
What security does the connecting program user need to have to run the Replication program?
- 3.
What Java libraries would you install in an Eclipse IDE Classpath to support the Replication program and where would you find them?
In Chapter 5, we will cover the use of the IBM Cognos BI Analytics, Real-Time Monitor to create a new Custom View for a Multilevel Time Dimension.
Note: Although Cognos BI 10.2.2 is at EOS as of December 31, 2022 (www.ibm.com/support/pages/end-continuing-support-cognos-business-intelligence-102x-effective-december-31-2022), the following IBM link announcement indicates that it is included in continuing support:
www.ibm.com/support/pages/cognos-analytics-continuing-support