Micro Focus Content Manager SDK 9.3
|
The BulkDataLoader object is designed to provide a high-performance method of loading records (and a few other associated things) into a TRIM database. The intention is to run this component inside a WorkgroupServer to avoid network RCF calls, and also to allow us to run at the sort of privilege where we can use the inherent DBMS bulk loading tools, etc. More...
Public Member Functions | |
BulkDataLoader (Origin forOrigin) | |
Constructs a BulkDataLoader component based on the supplied Origin object. More... | |
bool | Initialise () |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. It will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). Returns true for success. The ErrorMessage property should be checked when the return value is false. More... | |
bool | Initialise (bool useBulkLoadMode) |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. If you specify to use the fast "bulk loading" mode, it will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). Returns true for success. The ErrorMessage property should be checked when the return value is false. More... | |
bool | Initialise (bool useBulkLoadMode, bool relaxBusinessRulesForPerformance, bool relaxRulesForConversion) |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. If you specify to use the fast "bulk loading" mode, it will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). You can also specify that you wish to turn off certain intrinsic Content Manager behaviour to spped up performance (eg this will turn off some duplicate checking) or you can turn off other business rules that can be a problem when running imports as part of data conversion. Returns true for success. The ErrorMessage property should be checked when the return value is false. More... | |
TrimURI | StartRun (string nameOfActualSourceData) |
Tells the bulk loader that a run is about to commence. Allows you to specify the actual source object being used, and creates a history row. Returns the Uri allocated to this history run. More... | |
TrimURI | StartRun (string nameOfActualSourceData, string workingFolder) |
Tells the bulk loader that a run is about to commence. Allows you to specify the actual source object being used, as well as an alternate working directory into which the .DAT BULK INSERT files will be created and then subsequently read by the SQL server instance, so the path specified must be accessible to the SQL host machine. If this method is used, then the streaming of the bulkloader data files by the Workgroup Server is disabled. This will also create a history row. Returns the Uri allocated to this history run. More... | |
Record | NewRecord () |
Creates a new record for submission to the bulk loader. You should then set its document and then its properties using the SetDocument and SetProperties functions and any other required setup code. Once the record has been prepared it is submitted to the loader process using the SubmitRecord method. If this method returns a null item, check the ErrorMessage property. More... | |
Record | NewRecord (RecordType recordType) |
Creates a new record for submission to the bulk loader. You should then set its document and then its properties using the SetDocument and SetProperties functions and any other required setup code. Once the record has been prepared it is submitted to the loader process using the SubmitRecord method. If this method returns a null item, check the ErrorMessage property. This version allows you to specify the record type you wish to create. More... | |
bool | SetDocument (Record ofNewRecord, InputDocument electronicRecord, BulkLoaderCopyMode copyStyle) |
Sets the electronic document for the new TRIM record on the supplied TrimObject using input document provided. If the method returns false, you can check the ErrorMessage property for more details. See further comments for each of the BulkCopyStyle options. More... | |
bool | SetDocument (Record ofNewRecord, InputDocument electronicRecord) |
Calls SetDocument with a BulkCopyStyle of WindowsFileCopy More... | |
bool | SetProperties (TrimMainObject ofNewObject, PropertyOrFieldValueList objectMetadata) |
Sets the properties on the supplied TrimObject using the array of property values supplied. If the method returns false, you can check the ErrorMessage property for more details. More... | |
void | AllocateContainer (Record forNewRecord) |
Sets the container for the new record based on the Origin container allocation algorithm. If the property ContainerCreation is set to true for the Origin object, then the container will automatically be allocated when you call SubmitRecord. If you need access to the record's allocated container prior to submitting the record, then you can do the container allocation explicitly by calling this method. More... | |
TrimURI | SubmitRecord (Record newRecordToAdd) |
Adds the new record to the data loader input files. At this stage, the record is not created in TRIM - the actual processing is performed in the ProcessAccumulatedData function. Returns the uri that will be allocated to the new record when it is finally created. If the function returns 0, look at the ErrorMessage property for further details of the error. More... | |
bool | UpdateDocument (TrimURI recordUri, InputDocument electronicRecord, BulkLoaderCopyMode copyStyle) |
Adds a document update transaction into the set of data loader input files. At this stage, the record is updated in TRIM - the actual processing is performed in the ProcessAccumulatedData function. If the function returns false, look at the ErrorMessage property for further details of the error. See further comments for each of the BulkCopyStyle options. More... | |
bool | UpdateDocument (TrimURI recordUri, InputDocument electronicRecord) |
Calls UpdateDocument with a BulkCopyStyle of WindowsFileCopy More... | |
TrimURI | FindRecord (string withRecordId) |
Use mainly to check to see if a record with the specified record number exists. It is faster to use this method than the standard FindTrimObjectByName function, although still costs a server round trip. Returns the unique identifier of the matching record, or 0 if no match is found. More... | |
TrimURI | FindLocation (PropertyValue withLocationPropertyValue) |
Find a location that has the matching property value. The property values supported are : LocationSortName, LocationIdNumber, LocationEAddressName The function returns the uri of the location that matches. If no location matches the supplied value, a value of zero is returned. In addition, if there is more than one matching location, a value of zero is returned. More... | |
TrimURIList | FindAllLocations (PropertyValue withLocationPropertyValue) |
Finds all locations that are not people and match the supplied LocationSortName property. The function returns an array of uris for all matching locations. If no matches are found, the function will return null. More... | |
TrimURI | FindPerson (string surname, string givenNames, string initials, TrimDateTime dateOfBirth) |
Finds a person that matches the supplied parameters. Leave any of the parameters blank if you do not want to match on that property. The function returns the uri of the person that matches. If no person matches the supplied criteria, a value of zero is returned. In addition, if there is more than one matching person, a value of zero is returned. More... | |
TrimURIList | FindAllPeople (string surname, string givenNames, string initials, TrimDateTime dateOfBirth) |
Finds all people that match the supplied parameters. Leave any of the parameters blank if you do not want to match on that property. More... | |
Location | NewLocation () |
We recommended you use the NewLocation(LocationType) form of this function. More... | |
Location | NewLocation (LocationType typeOfLocation) |
Creates a new location of the specified type for submission to the bulk loader. You should then set its properties using the UpdateProperties function and any other required setup code. Once the location has been prepared it is submitted to the loader process using the SubmitLocation method. If this method returns a null item, check the ErrorMessage property. More... | |
TrimURI | SubmitLocation (Location newLocationToAdd) |
Adds the new location to the data loader input files. At this stage, the location is not created in TRIM - the actual processing is performed in the ProcessAccumulatedData function. Returns the uri that will be allocated to the new location when it is finally created. If the function returns 0, look at the ErrorMessage property for further details of the error. More... | |
void | ProcessAccumulatedData () |
This function processes all the data that has been supplied by calls to AddRecord, AddLocation, etc. The BulkDataLoader formats input files suitable for bulk loading into the various SQL engines - this is the function that passes these files off to the SQL engine for processing. More... | |
void | EndRun () |
This function indicates that this import run is complete, any outstanding batches will be processed, the run history row will be updated for a final time and events will be generated to schedule the text and content indexing of any records created. More... | |
Public Member Functions inherited from HP.HPTRIM.SDK.TrimSDKObject | |
TrimSDKObject () | |
Properties | |
bool | CreateContentIndexEvents [get, set] |
Gets or sets a boolean value indicating whether the bulk loader should notify the event processor to index the content related to a batch of records. You can create this event notification later using the TRIM User interface if you wish to defer the indexing work. More... | |
bool | DoWordIndexing [get, set] |
Gets or sets a boolean value indicating whether the bulk loader should perform word indexing for imported records. If word indexing is suppressed, then the records will need to be indexed manually later. This can be done using the CM User interface and selecting the appropriate option for the origin history. More... | |
bool | CreateWordIndexEvents [get, set] |
Gets or sets a boolean value indicating how the bulk loader should do word indexing. The records can be indexed by generating an event for the event processor, or directly indexed as part of the bulk loader processAccumluatedData command. If the DoWordIndexing property is set to false, then this property will have no effect. More... | |
TrimURI | RunHistoryUri [get] |
Gets the run history object associated with this run. This object is created by the StartRun function. More... | |
string | LogFileName [get, set] |
Specify the name of a log file where the bulk loader will write diagnostic information in the event that the update to the database fails. If there are no errors during the update, then this file will not be written to and generally, it should not contain anything. The accepted method of error checking during the bulk loading process is to verify the return values of all methods and to check the ErrorMessage property if something unexpected occurs. Set this property before you call StartRun. More... | |
bool | UseOracleDirectPathLoad [get, set] |
Gets or sets a boolean value indicating whether to use the Oracle Direct Path Load feature when loading data using the Oracle SQLLDR utility. TYpically you would use this if the load is being run while TRIM users are not actively using the system and there is no other concurrent record update activity happening. More... | |
bool | SQLServerTableLock [get, set] |
Gets or sets a boolean value indicating whether to use the TABLOCK command line parameter for the SQL server BULK INSERT command. TABLOCK can improve performance however may cause other user updates to slow down. Refer to the SQL Server documentation for BULK INSERT for further information. More... | |
bool | SQLCheckConstraints [get, set] |
Gets or sets a boolean value indicating whether to enable the "CHECK CONSTRAINTS" directive during BULK INSERT operations. There is a performance hit when enabling this setting, but it does ensure that table constraints are honoured. The default setting of false can mean that certain unique indexes on tables are not enforced and therefore will need to be fixed manually. Refer to the SQL Server documentation for BULK INSERT for further information. More... | |
bool | UseBulkLoaderRecordNumbering [get, set] |
Gets or sets a boolean value indicating whether records should use the special BULK loader numbering system, or whether to use the numbering provided with the record type. It should be noted that the Record Type numbering options involves a round trip to the workgroup server and so will not result in the best performance. Default value is true. More... | |
bool | StreamDataToWorkgroup [get, set] |
Gets or sets a boolean value indicating whether the intermediate data files need to be streamed up to the workgroup server, or whether the database server is capable of processing them directly. More... | |
bool | AutoCommitLocations [get, set] |
Gets or sets a boolean value indicating whether Locations should be processed immediately. When using the bulkloader for creating locations, it is often difficult to use these locations because the bulkloader approach means the location details are not available for verification purposes until after the batch commit step, which is a call to the ProcessAccumulatedData() method. In particular, using imported locations as: Record Assignee, Record Home or Record Owner, require a location that exists in the database. For a value of true, Locations will be saved to the database as soon as they are encountered, i.e. during the SubmitLocation() method. Searching for the Location will return a result, thus avoiding a duplicate Location from being created. This turns bulkloading off for Locations and the Location is processed in the same way as it would be using the standard SDK methods to create a Location. If set to false (default), Locations will be saved to the database when the batch is processed, i.e. after a call to ProcessAccumulatedData(). Searching for the Location prior to this will not yield a successful result. More... | |
bool | ExtractMetadataFromContent [get, set] |
Gets or sets a boolean value indicating whether content should be examined for metadata when attaching a document to a new record. Basic metadata is obtained from the document title and creations date, however you can get further information such as document properties for openXML docs or EXIF data from JPG files if you enable this option. More... | |
bool | CreateContactsFromEmailRecipients [get, set] |
Gets or sets a boolean value indicating whether contacts should be created from email recipients. More... | |
bool | CreateContactsOnlyForExistingLocations [get, set] |
Gets or sets a boolean value indicating when creating contacts from email recipients, limit contacts to only locations that already exist in the dataset. This prevents creating new 'unknown' locations from email addresses that haven't already been registered for a location. More... | |
Properties inherited from HP.HPTRIM.SDK.TrimSDKObject | |
string | ErrorMessage [get] |
All TRIM SDK objects have an ErrorMessage property which can be used to examine any error that may have occurred in the most recent method call. More... | |
TrimException | Error [get] |
All TRIM SDK objects have an Error property which can be used to examine any error that may have occurred in the most recent method call. More... | |
The BulkDataLoader object is designed to provide a high-performance method of loading records (and a few other associated things) into a TRIM database. The intention is to run this component inside a WorkgroupServer to avoid network RCF calls, and also to allow us to run at the sort of privilege where we can use the inherent DBMS bulk loading tools, etc.
HP.HPTRIM.SDK.BulkDataLoader.BulkDataLoader | ( | Origin | forOrigin | ) |
Constructs a BulkDataLoader component based on the supplied Origin object.
void HP.HPTRIM.SDK.BulkDataLoader.AllocateContainer | ( | Record | forNewRecord | ) |
Sets the container for the new record based on the Origin container allocation algorithm. If the property ContainerCreation is set to true for the Origin object, then the container will automatically be allocated when you call SubmitRecord. If you need access to the record's allocated container prior to submitting the record, then you can do the container allocation explicitly by calling this method.
void HP.HPTRIM.SDK.BulkDataLoader.EndRun | ( | ) |
This function indicates that this import run is complete, any outstanding batches will be processed, the run history row will be updated for a final time and events will be generated to schedule the text and content indexing of any records created.
TrimURIList HP.HPTRIM.SDK.BulkDataLoader.FindAllLocations | ( | PropertyValue | withLocationPropertyValue | ) |
Finds all locations that are not people and match the supplied LocationSortName property. The function returns an array of uris for all matching locations. If no matches are found, the function will return null.
TrimURIList HP.HPTRIM.SDK.BulkDataLoader.FindAllPeople | ( | string | surname, |
string | givenNames, | ||
string | initials, | ||
TrimDateTime | dateOfBirth | ||
) |
Finds all people that match the supplied parameters. Leave any of the parameters blank if you do not want to match on that property.
TrimURI HP.HPTRIM.SDK.BulkDataLoader.FindLocation | ( | PropertyValue | withLocationPropertyValue | ) |
Find a location that has the matching property value. The property values supported are : LocationSortName, LocationIdNumber, LocationEAddressName The function returns the uri of the location that matches. If no location matches the supplied value, a value of zero is returned. In addition, if there is more than one matching location, a value of zero is returned.
TrimURI HP.HPTRIM.SDK.BulkDataLoader.FindPerson | ( | string | surname, |
string | givenNames, | ||
string | initials, | ||
TrimDateTime | dateOfBirth | ||
) |
Finds a person that matches the supplied parameters. Leave any of the parameters blank if you do not want to match on that property. The function returns the uri of the person that matches. If no person matches the supplied criteria, a value of zero is returned. In addition, if there is more than one matching person, a value of zero is returned.
TrimURI HP.HPTRIM.SDK.BulkDataLoader.FindRecord | ( | string | withRecordId | ) |
Use mainly to check to see if a record with the specified record number exists. It is faster to use this method than the standard FindTrimObjectByName function, although still costs a server round trip. Returns the unique identifier of the matching record, or 0 if no match is found.
bool HP.HPTRIM.SDK.BulkDataLoader.Initialise | ( | ) |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. It will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). Returns true for success. The ErrorMessage property should be checked when the return value is false.
bool HP.HPTRIM.SDK.BulkDataLoader.Initialise | ( | bool | useBulkLoadMode | ) |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. If you specify to use the fast "bulk loading" mode, it will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). Returns true for success. The ErrorMessage property should be checked when the return value is false.
bool HP.HPTRIM.SDK.BulkDataLoader.Initialise | ( | bool | useBulkLoadMode, |
bool | relaxBusinessRulesForPerformance, | ||
bool | relaxRulesForConversion | ||
) |
Initialise the environment prior to running any bulk loading operations. This method should be called directly after constructing a BulkDataLoader object. If you specify to use the fast "bulk loading" mode, it will search for a database connection string as well as database utilities required for bulk inserting (for example Oracle SQLLDR.exe). You can also specify that you wish to turn off certain intrinsic Content Manager behaviour to spped up performance (eg this will turn off some duplicate checking) or you can turn off other business rules that can be a problem when running imports as part of data conversion. Returns true for success. The ErrorMessage property should be checked when the return value is false.
Location HP.HPTRIM.SDK.BulkDataLoader.NewLocation | ( | ) |
We recommended you use the NewLocation(LocationType) form of this function.
Location HP.HPTRIM.SDK.BulkDataLoader.NewLocation | ( | LocationType | typeOfLocation | ) |
Creates a new location of the specified type for submission to the bulk loader. You should then set its properties using the UpdateProperties function and any other required setup code. Once the location has been prepared it is submitted to the loader process using the SubmitLocation method. If this method returns a null item, check the ErrorMessage property.
Record HP.HPTRIM.SDK.BulkDataLoader.NewRecord | ( | ) |
Creates a new record for submission to the bulk loader. You should then set its document and then its properties using the SetDocument and SetProperties functions and any other required setup code. Once the record has been prepared it is submitted to the loader process using the SubmitRecord method. If this method returns a null item, check the ErrorMessage property.
Record HP.HPTRIM.SDK.BulkDataLoader.NewRecord | ( | RecordType | recordType | ) |
Creates a new record for submission to the bulk loader. You should then set its document and then its properties using the SetDocument and SetProperties functions and any other required setup code. Once the record has been prepared it is submitted to the loader process using the SubmitRecord method. If this method returns a null item, check the ErrorMessage property. This version allows you to specify the record type you wish to create.
void HP.HPTRIM.SDK.BulkDataLoader.ProcessAccumulatedData | ( | ) |
This function processes all the data that has been supplied by calls to AddRecord, AddLocation, etc. The BulkDataLoader formats input files suitable for bulk loading into the various SQL engines - this is the function that passes these files off to the SQL engine for processing.
bool HP.HPTRIM.SDK.BulkDataLoader.SetDocument | ( | Record | ofNewRecord, |
InputDocument | electronicRecord, | ||
BulkLoaderCopyMode | copyStyle | ||
) |
Sets the electronic document for the new TRIM record on the supplied TrimObject using input document provided. If the method returns false, you can check the ErrorMessage property for more details. See further comments for each of the BulkCopyStyle options.
bool HP.HPTRIM.SDK.BulkDataLoader.SetDocument | ( | Record | ofNewRecord, |
InputDocument | electronicRecord | ||
) |
Calls SetDocument with a BulkCopyStyle of WindowsFileCopy
bool HP.HPTRIM.SDK.BulkDataLoader.SetProperties | ( | TrimMainObject | ofNewObject, |
PropertyOrFieldValueList | objectMetadata | ||
) |
Sets the properties on the supplied TrimObject using the array of property values supplied. If the method returns false, you can check the ErrorMessage property for more details.
TrimURI HP.HPTRIM.SDK.BulkDataLoader.StartRun | ( | string | nameOfActualSourceData | ) |
Tells the bulk loader that a run is about to commence. Allows you to specify the actual source object being used, and creates a history row. Returns the Uri allocated to this history run.
TrimURI HP.HPTRIM.SDK.BulkDataLoader.StartRun | ( | string | nameOfActualSourceData, |
string | workingFolder | ||
) |
Tells the bulk loader that a run is about to commence. Allows you to specify the actual source object being used, as well as an alternate working directory into which the .DAT BULK INSERT files will be created and then subsequently read by the SQL server instance, so the path specified must be accessible to the SQL host machine. If this method is used, then the streaming of the bulkloader data files by the Workgroup Server is disabled. This will also create a history row. Returns the Uri allocated to this history run.
Adds the new location to the data loader input files. At this stage, the location is not created in TRIM - the actual processing is performed in the ProcessAccumulatedData function. Returns the uri that will be allocated to the new location when it is finally created. If the function returns 0, look at the ErrorMessage property for further details of the error.
Adds the new record to the data loader input files. At this stage, the record is not created in TRIM - the actual processing is performed in the ProcessAccumulatedData function. Returns the uri that will be allocated to the new record when it is finally created. If the function returns 0, look at the ErrorMessage property for further details of the error.
bool HP.HPTRIM.SDK.BulkDataLoader.UpdateDocument | ( | TrimURI | recordUri, |
InputDocument | electronicRecord, | ||
BulkLoaderCopyMode | copyStyle | ||
) |
Adds a document update transaction into the set of data loader input files. At this stage, the record is updated in TRIM - the actual processing is performed in the ProcessAccumulatedData function. If the function returns false, look at the ErrorMessage property for further details of the error. See further comments for each of the BulkCopyStyle options.
bool HP.HPTRIM.SDK.BulkDataLoader.UpdateDocument | ( | TrimURI | recordUri, |
InputDocument | electronicRecord | ||
) |
Calls UpdateDocument with a BulkCopyStyle of WindowsFileCopy
|
getset |
Gets or sets a boolean value indicating whether Locations should be processed immediately. When using the bulkloader for creating locations, it is often difficult to use these locations because the bulkloader approach means the location details are not available for verification purposes until after the batch commit step, which is a call to the ProcessAccumulatedData() method. In particular, using imported locations as: Record Assignee, Record Home or Record Owner, require a location that exists in the database. For a value of true, Locations will be saved to the database as soon as they are encountered, i.e. during the SubmitLocation() method. Searching for the Location will return a result, thus avoiding a duplicate Location from being created. This turns bulkloading off for Locations and the Location is processed in the same way as it would be using the standard SDK methods to create a Location. If set to false (default), Locations will be saved to the database when the batch is processed, i.e. after a call to ProcessAccumulatedData(). Searching for the Location prior to this will not yield a successful result.
|
getset |
Gets or sets a boolean value indicating whether contacts should be created from email recipients.
|
getset |
Gets or sets a boolean value indicating when creating contacts from email recipients, limit contacts to only locations that already exist in the dataset. This prevents creating new 'unknown' locations from email addresses that haven't already been registered for a location.
|
getset |
Gets or sets a boolean value indicating whether the bulk loader should notify the event processor to index the content related to a batch of records. You can create this event notification later using the TRIM User interface if you wish to defer the indexing work.
|
getset |
Gets or sets a boolean value indicating how the bulk loader should do word indexing. The records can be indexed by generating an event for the event processor, or directly indexed as part of the bulk loader processAccumluatedData command. If the DoWordIndexing property is set to false, then this property will have no effect.
|
getset |
Gets or sets a boolean value indicating whether the bulk loader should perform word indexing for imported records. If word indexing is suppressed, then the records will need to be indexed manually later. This can be done using the CM User interface and selecting the appropriate option for the origin history.
|
getset |
Gets or sets a boolean value indicating whether content should be examined for metadata when attaching a document to a new record. Basic metadata is obtained from the document title and creations date, however you can get further information such as document properties for openXML docs or EXIF data from JPG files if you enable this option.
|
getset |
Specify the name of a log file where the bulk loader will write diagnostic information in the event that the update to the database fails. If there are no errors during the update, then this file will not be written to and generally, it should not contain anything. The accepted method of error checking during the bulk loading process is to verify the return values of all methods and to check the ErrorMessage property if something unexpected occurs. Set this property before you call StartRun.
|
get |
Gets the run history object associated with this run. This object is created by the StartRun function.
|
getset |
Gets or sets a boolean value indicating whether to enable the "CHECK CONSTRAINTS" directive during BULK INSERT operations. There is a performance hit when enabling this setting, but it does ensure that table constraints are honoured. The default setting of false can mean that certain unique indexes on tables are not enforced and therefore will need to be fixed manually. Refer to the SQL Server documentation for BULK INSERT for further information.
|
getset |
Gets or sets a boolean value indicating whether to use the TABLOCK command line parameter for the SQL server BULK INSERT command. TABLOCK can improve performance however may cause other user updates to slow down. Refer to the SQL Server documentation for BULK INSERT for further information.
|
getset |
Gets or sets a boolean value indicating whether the intermediate data files need to be streamed up to the workgroup server, or whether the database server is capable of processing them directly.
|
getset |
Gets or sets a boolean value indicating whether records should use the special BULK loader numbering system, or whether to use the numbering provided with the record type. It should be noted that the Record Type numbering options involves a round trip to the workgroup server and so will not result in the best performance. Default value is true.
|
getset |
Gets or sets a boolean value indicating whether to use the Oracle Direct Path Load feature when loading data using the Oracle SQLLDR utility. TYpically you would use this if the load is being run while TRIM users are not actively using the system and there is no other concurrent record update activity happening.