• Service Expression Name: harvest resource—oai-pmh lom ws
• e-Framework Service Expression Registry Name: TBD

• Service Genre Name: harvest
• e-Framework Service Genre Registry Name: TBD
  

• Version: v0.1
• e-Framework Service Expression Version: TBD

• Author name(s) : Dan Rehak and Nick Nicholas
• Organisation : Federated Repositories for Education (FRED)

This service expression is a specialization of the harvest service genre. It uses the OAI-PMH specification for harvesting metadata. All of the OAI-PMH operators (verbs) are supported. The resource (provider repository) being harvested exposes an interface for harvesting. Communications to the provider service implementation interface are represented in requests defined via web services, using SOAP over HTTP transport through the SOAP HTTP binding. Requests and (dissemination) results are communicated between the client/requestor and the provider service implementation using HTTP, with results encoded in XML. The service expression supports disseminating LOM v1.0 metadata from the resource. The service implementation does not include a mechanism to authorize clients. How to discover service implementations that support harvesting is out of scope.

The service expression description that follows uses e-Framework service expression description elements as of 2006-12-09. Other terms, e.g., client, provider, resource, are used as defined in the e-Framework. These definitions may conflict with those used in the OAI-PMH specification.

Items are tagged and identified using names assigned by the FRED project. Formal e-Framework names will be assigned by the e-Framework.

The harvest resource—oai-pmh lom ws service expression specializes the harvest service genre through binding to the OAI-PMH specification and use of a SOAP-based web services binding. It further constrains the service implementations with additional constraints that are specific to the FRED community.

Provider repositories expose a “harvest” interface, defined by this service expression as a web service. Clients may send requests to this interface to gather the metadata stored in the repository. The OAI-PMH specification details some of the requirements on such an interface. This service expression follows OAI-PMH. Communications between the clients that harvest metadata from the repository and service implementation interface (repository interface) are through web services, not HTTP requests. Repositories are required to expose Learning Object Metadata (LOM) v1.0 metadata for objects in the repository. This service expression places additional requirements specific to the FRED community on all service implementations with the purpose of increasing interoperability.

Repository harvest interfaces are not access controlled, i.e., any client may attempt to harvest the metadata. There are no authentication controls. The provider is responsible for determining what results it will return. How a repository and service implementation decide how to respond to requests (i.e., what they expose) is not specified. The service expression only details the format and encoding of requests and responses, and specifics constraints and requirements on service implementations.

As specified by OAI-PMH, the harvest resource service expression supports six distinct functions.

• Identify: Get descriptive information about the repository (the resource behind the service implementation interface) and the capabilities of the service implementation. This information enables a client to successfully communicate with the repository. Such information includes the repository’s treatment of deleted records, and the granularity of its date stamps.
• ListMetadataFormats: Get the set of metadata formats that the repository is capable of providing as results for a harvest request. The repository may be capable of returning (disseminating) the metadata in different formats. This information enables a client to request metadata in a particular format
• ListSets: Get information about the structure of the repository. Objects in the repository may be grouped into “sets”. This information enables a client to selectively harvest data by specifying the parts of the repository to be harvested
• GetRecord: Get the metadata for a single specific object in the repository. The client may request the desired metadata format. This function enables the client to get the metadata for a single item.
• ListRecords: Get metadata records from the repository. This function enables the client to get the metadata for a selected collection of items. The client may request selective harvesting by specifying a date range or by specifying a part of the repository (via "sets"). Metadata records are returned for the objects that match the selection criteria. Mechanisms exist to provide flow control so the results may be returned in chunks.
• ListIdentifiers: Get the the headers of metadata records in the repository (record identifier, date stamp, sets the record belongs to, deletion status). This function enables the client to get summary information for a selected collection of items. The client may request selective results by specifying a date range or by specifying a part of the repository (via “sets”). Mechanisms exist to provide flow control so the results may be returned in chunks.

The service expression provides the capability to return LOM metadata records for objects. A service implementation MAY support other metadata formats. In order to remain compliant with OAI-PMH, a service implementation MUST support Dublin Core..

Communications to the service implementation are via web services, using SOAP over HTTP transport, and the SOAP HTTP binding.

No other functionality is provided. The functionality defined SHALL NOT be extended.

Harvesting occurs to facilitate discovery of assets in the provider repository: discovery is transacted on the client repository instead, by means of the metadata harvested. The main reason to do this is when a single client repository gathers metadata from multiple provider repositories. In that case, discovery across the range of repositories involves only a single query on the client, rather than a separate query for each provider. The providers form a federation of repositories by having their metadata harvested and centralised. The federation can be ad hoc, with metadata harvested as found on open access repositories; or it can be formal, with policy guarantees such as uniform metadata profiles and service level agreements underpinning it. The typical scenario is:

1. Client harvests metadata instance for an item from the provider repository
2. Client validates the submitted metadata against the registry schema
3. Metadata is associated with content item

The associate step is necessary only if the identifier used at the client repository to refer to content is distinct from that used at the provider repository. If both repositories use the same global identifier, it is skipped.

Once a metadata record is discovered on the client through a query, the user may gains access to the content item referred to by the metadata record. This access occurs through a locator or identifier inside the metadata record harvested from the provider, possibly converted to the client’s namespace through the associate step. The identifier for content is logically distinct from the identifier for the metadata record referring to it; the metadata record identifier is the parameter used by OAI-PMH. The content item need not reside on the provider repository.

The metadata on the client repository is expected to stay reasonably up to date. As a result, harvesting will usually be a periodically recurrent activity, and restricted only to those metadata records updated on the provider repository since the last harvesting session.

A client may not be interested in ingesting all the metadata records available from the provider, but only those relevant to the client repository subject matter. The provider offers a mechanism of restricting harvesting to those records identified as relevant; in OAI-PMH, this is done through “sets”.

Harvesting proper is done through the ListRecords function. Of the other functions, Identify and ListMetadataFormats provide the client with the information necessary to set harvesting up, and choose the metadata profile relevant. (This service expression presumes LOM has already been confirmed as an available metadata format.) ListSets allows the client to restrict harvesting to the subsets of the provider repository contents it finds relevant. The scenario outlined fetches metadata as a batch rather than as individual records (GetRecord). It gathers complete metadata records, including local identifiers, in order to allow discovery on the client repository. The alternative is that it fetches only the headers of a set of records, e.g. recently updated records (ListIdentifiers), and either fetches the full records piecemeal, or makes the user browse them on the provider repository.

So, integrating these in on overall workflow:

A client acting on behalf of a federation of vocational education repositories harvests metadata from a participating provider.

• What version of OAI PMH does the provider support, and how does it handle date stamps and deleted records? (Identify)
• Given that: does the provider support LOM? (ListMetadataFormats)
• Given that: does the provider allow selective harvesting of only records describing vocational education modules? (ListSets; this presupposes an agreed vocabulary of sets, and that the provider may contain items other than vocational education)
• Alt 1: Request all headers of relevant metadata records. This is summary information about the metadata records, including when records have been updated and what the identifiers for the records are. The client may retrieve the corresponding records later, or redirect the end user to those metadata records on the provider. (ListIdentifiers)
• Alt 2: Request all relevant metadata records. These will be ingested into the client repository (ListRecords)
• Alt 3: Given the identifier for a metadata record (which may have been retrieved though ListIdentifiers), request the full corresponding metadata record. (GetRecord)

As defined, the service expression harvests LOM metadata from repositories without access controls.

The service expression MAY be extended (by specifying the metadata format) if harvesting of other metadata formats is required. The Dublin Core metadata format MUST be provided for OAI-PMH conformance. The form of any extension to a new metadata format (other than Dublin Core) is outside the scope of this document, and will constitute a new service expression.

The service expression is not applicable when the repository requires authentication to permit harvest. (NB: Such extensions require a new service expression.)

The service expression is not applicable when the repository filters the results that are returned based on authorization policies or rules that need to be communicated to the repository through the harvest interface. (NB: Such extensions require a new service expression.)

The service expression is not applicable if communications need to be secure. (NB: Such extensions require a new service expression.)

The format for request and responses are defined in the OAI-PMH specification. Six requests (verbs) SHALL be supported; these are the same requests enumerated under Functionality.

Identify (See OAI-PMH specification clause 4.2). The request and response SHALL be as specified. Additional requirements:

• The encoded repositoryName value (giving the human-readable name of the provider repository) SHALL NOT exceed 255 bytes in length.
• The protocolVersion value (version of OAI-PMH) SHALL be 2.0
• Any encoded adminEmail value (contact email of provider administrator) SHALL NOT exceed 255 bytes in length.
• The granularity value (minimum granularity of date stamp) SHALL be the finest granularity supported by the service implementation.

ListMetadataFormats (See OAI-PMH specification clause 4.4). The request and response SHALL be as specified. Additional requirements:

• The service implementation SHALL support dissemination of LOM v1.0. The description of the metadataFormat SHALL be

ListSets (See OAI-PMH specification clause 4.6). The request and response SHALL be as specified. Additional requirements:

• The encoded resumptionToken value SHALL NOT exceed 255 bytes in length.

GetRecord (See OAI-PMH specification clause 4.1). The request and response SHALL be as specified. Additional requirements:

• The encoded identifier value SHALL NOT exceed 255 bytes in length.

ListRecords (See OAI-PMH specification clause 4.5). The request and response SHALL be as specified. Additional requirements:

• The encoded resumptionToken value SHALL NOT exceed 255 bytes in length.

ListIdentifiers (See OAI-PMH specification clause 4.3). The request and response SHALL be as specified. Additional requirements:

• The encoded identifier value SHALL NOT exceed 255 bytes in length.
• The encoded resumptionToken value SHALL NOT exceed 255 bytes in length.

The model for a client to interact with a service implementation is defined in the OAI-PMH specification; see under Usage Scenarios for typical use of harvesting. Except as specified herein, the service implementation SHALL conform to the OAI-PMH specification.

The service implementation SHALL implement flow control (See OAI-PMH specification clause 3.5) with the following additional requirements:

• Flow control SHALL NOT be used if the number of records in the response is less than 1001. All records SHALL be returned in a single result set.
• The number of records in the results set SHALL NOT exceed 1000.
• The resumptionToken SHALL include an expirationDate. The specified expiration date of the resumptionToken SHALL be at least 10 minutes from the time the service implementation transmits the response to the client.

Responses for all requests SHALL include all applicable error codes. OAI-PMH errors SHALL be communicated in the OAI_PMH response, as specified in the result schema, while SOAP errors SHALL be handled at the SOAP protocol level.

The data model for OAI-PMH harvests is given in the OAI-PMH specification §2. In brief:

• Repositories contain metadata describing content items (in OAI-PMH resources), which may or may not be stored in the same repositories.
• These metadata descriptions are stored or generated by items which reside on the repository.
• These metadata descriptions are stored or generated in the form of records.
• Repositories support one or more metadata formats.
• There may be several metadata records per item—i.e. each content item may be described by several metadata records. These records can differ from each other only in metadata format: an item can have at most one metadata record for each supported metadata format.
• Items are identified by unique identifiers.
• Therefore, the combination of identifier and metadata format identifies a unique metadata record.
• Items can be assigned to sets, which enable selective harvesting.
• Records are returned as XML; as specified in the OAI-PMH XML Schema, this contains:
  • A header, which in turn contains:
    • The item identifier
    • The datestamp for the record
    • The set membership of the item
    • An optional status attribute, indicating that the item has been deleted from the repository
  • The metadata record proper
  • An optional about container of data, which is metadata about the metadata—including e.g. a rights statement for the metadata record, or a description of the provenance of the metadata record

All OAI-PMH requests are read-only, and do not change the state of the provider repository. The List requests (ListRecords, ListIdentifiers, ListSets) allow for the possibility that the requested data may be too large to return in a single response. Instead of a single request query, OAI-PMH allows a List transaction to consist of multiple request–response pairs; each incomplete response will contain a resumption token, which the client must invoke to obtain the next segment of the response. The provider must therefore treat List requests as stateful: it generates a listing of sets or items in response to the query (and extracts identifiers or records from the items in turn), but it must remember how many sets or items it has already presented to the client. Resumption tokens have optional expiration dates, so the provider MAY need to commit to holding state information on List requests for a significant period of time.

Between queries, the set of records that the initial request specified may change, as new records are created, modified or deleted; the provider MAY return an error indicating that the request should be reinitiated. If the request is honoured, the provider MUST return all unchanged records. It MAY update its listing of sets or items to reflect the intervening changes, excluding or including changed records.

Providers are also expected to notify clients if any individual metadata records have been deleted (or become unavailable), even if the item generating them remains in the provider repository. If a provider discontinues LOM support, its LOM metadata records are deemed deleted, though its Dublin Core records for the same items remain in place.

Selective harvesting of records may be parameterised on date stamp, so that the client limits its request to those records created, modified or deleted within the given date span. A compliant provider MUST commit to keeping date stamps for all changes to the item resulting in an altered metadata record, and to exposing the mapping between date stamp and record (i.e. an index of items by most recent date stamp). This includes all creation and deletion of items, and all updates in metadata values, as well as all changes in metadata format (which will result in a different metadata record even if the content of the record is unchanged). Providers MAY have a date granularity of seconds rather than days.

Selective harvesting of records may be parameterized by set; this means that the provider must expose a mapping between set and record (i.e. an index of items by the set(s) they have been associated with).

The service expression interface SHALL conform to the FRED Profile for Core Service Standards

There is no WSDL currently defined as a standard for OAI-PMH: OAI-PMH is defined only as an HTTP-transport protocol. Absent an official WSDL, a FRED OAI-PMH WDSL SHALL be used. The FRED WDSL is based on the IVOA WSDL ( http://www.ivoa.net/internal/IVOA/RegistryInterface/RegistryHarvest-v1.0.wsdl ), and is hosted at [link]. It has the namespace [link]. Requests to OAI-PMH SHALL follow the format given in the WDSL.

The OAI-PMH response to a request SHALL be well-formed XML validating against the OAI-PMH XML Schema, as it is described in the OAI-PMH specification. The schema is available at: http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd . The schema allows a response to contain either data or an error report. Additional constraints on the sizes of XML elements in the result are defined in Behaviours & Requests elements within the service expression description. A base URL SHALL be specified as the content of the element in the OAI-PMH response. The base URL SHALL be the URI of the SOAP ultimate receiver node, expressed as a URL.

The XML schema for LOM metadata results SHALL be as defined in IEEE 1484.12.3-2005. The schema is available at: http://ltsc.ieee.org/xsd/lomv1.0/lom.xsd.

The service expression SHALL be implemented through SOAP HTTP binding, allowing the SOAP Request-Response message exchange pattern, which uses HTTP POST (SOAP v1.2 2 7.5). Illustration:

This XML response SHALL be encapsulated within the body of a SOAP envelope, which MAY exclude the XML declaration, as follows:

A service implementation SHALL use SOAP error codes rather than HTTP Status codes to flag either load balancing or service unavailable. The HTTP Status codes used in a service implementation SHALL otherwise be restricted to those specified for the SOAP HTTP binding (SOAP v.1.2 2 7.5).

• SOAP error processing SHALL generate an error message for each SOAP error encountered, rather than allowing only one random message to be generated.
• A VersionMismatch SOAP fault SHALL provide an Upgrade SOAP header block, detailing what SOAP envelope versions are supported by the node (SOAP v.1.2 1 5.4.7).
• A decoding SOAP fault of subcode enc:MissingID, enc:DuplicateID, or enc:UntypedValue SHALL be generated under the conditions given in SOAP v.1.2 2 3.2. A RPC SOAP fault of code env:Receiver, env:DataEncodingUnknown or env:Sender SHALL be generated under the conditions given in SOAP v.1.2 4.4.

Response compression SHALL be offered by service implementation through the SOAP HTTP binding, with the appropriate use of the HTTP Content-Encoding header field (SOAP v1.2 2 7.5.1.1), and complying with the constraints imposed by OAI-PMH (3.1.3). The service implementation SHALL support gzip compression.

The service expression SHALL accept URIs of at least 255 bytes in length. A SOAP Fault code with value Sender (indicating malformed message), and a detail element explaining the malformed URI, SHALL be generated if the URI is too large to be processed. All child elements of the SOAP envelope SHALL be namespace-qualified (SOAP v1.2. 1 5.3.1)

SOAP Active Intermediaries SHALL NOT be used in the context of this service expression (SOAP v1.2 1 2.7.3).

A service implementation SHALL return an HTTP status 400 BAD Request when it detects that the request contains code injection or other malicious elements.

HTTP requests SHALL indicate the ability to accept the MIME type application/soap+xml (SOAP v.1.2 2 7.1.4), and responses SHALL be of MIME type application/soap+xml. The HTTP charset parameter SHALL be used (as recommended by WS-I Basic Profile v.1.2 R1018), and takes priority over the XML declaration charset parameter, if present.

Consistency:

• The service implementation SHALL ensure that the time stamps for when content is added or modified are consistent with those returned to the client such that the metadata for all objects is harvestable.

Performance:

• A service implementation SHALL be capable of handling simultaneous requests from different clients.
• A service implementation SHOULD implement an indexing scheme or equivalent method to permit efficient harvesting by date ranges.
• Load balancing SHOULD be implemented for large repositories or those which are harvested frequently (continuously).
• The IEEE reference schema for LOM is multilayered and considered cumbersome; see for example http://www.ostyn.com/standards/scorm/samples/simplerlomschemadoc.htm. However, to guarantee conformance with any LOM instance, including custom extensions, the IEEE schema SHALL be used.
• The resumption token protocol of OAI-PMH means that a large result set must be harvested sequentially; so it is not parallelisable: a large result set may not be harvested by two harvesters in parallel, or out of sequence. The first harvesting of a large repository will therefore be time consuming, though it may not be time-intensive (resumption tokens may have a long time-to-live.)
• The harvest request to a set of repositories is trivially parallelisable: the same request to harvest may be made to several repositories at the same time. Ingesting several sets of metadata records returned simultaneously is a challenge for an Ingest service, but not a Harvest service.

Interoperability:

• Clients should be aware that URIs longer than 255 bytes may not be supported by intermediaries (caches, proxies).

Security:

• A service implementation SHALL inspect all requests for possible code injection.
• A client SHOULD validate all XML results against appropriate schemas. (SOAP v1.2 2 C).
• SOAP messages SHOULD validate against both the minimum schema and the SOAP Encoding schema.
• Only well-defined SOAP header and body blocks should be processed (SOAP v.1.2 1 7.1).
• Since SOAP intermediary nodes are men-in-the-middle, all messages must be properly authenticated with regard to all nodes along the message path (SOAP v1.2 1 7). See also SOAP v.1.2 2 A.2 for HTTP Binding, RFC 3023 on XML Media types, section 10; WS-I v1.2 section 6.

The Open Archives Initiative Protocol for Metadata Harvesting, Protocol Version 2.0. http://www.openarchives.org/OAI/openarchivesprotocol.html

Dublin Core Metadata Element Set, Version 1.1: Reference Description. http://dublincore.org/documents/dces/

IEEE Standard for Learning Object Metadata, IEEE Standards Department, Institute of Electrical and Electronic Engineers, Inc. (2002). IEEE-SA Standard 1484.12.1-2002. http://ltsc.ieee.org/wg12/

Hypertext Transfer Protocol -- HTTP/1.1. http://www.ietf.org/rfc/rfc2616.txt

FRED Profile for Core Service Standards [Handle to be supplied]

Security Considerations:

• Service implementation may be subject to denial-of-service attacks.
• Administrator email addresses are returned to the client. Requests include a From request-header field with an email address. Care should be taken to maintain privacy of email addresses.
• Service implementation may log request and results. Security of logs should be maintained.
• There are no authorization or authentication controls. Care should be taken to maintain data privacy.
• While a service implementation may return provenance data, clients are not obliged to process it.
• Service implementations must respect flow control responses from repositories, including HTTP error codes such as 503 Service Unavailable and 302 Found.

As pointed out by Congia et al., “The SOAP Header is not used since its definition suggests it is intended for intermediate or non-payload processing and the OAI-PMH essentially requires end-to-end payload delivery.” Accordingly, the routing etc. information that could potentially be included in the SOAP Header is outside the scope of this service expression, and a SOAP Header SHOULD NOT be used; Mandatory SOAP header blocks (mustUnderstand) SHALL NOT be used. SOAP Role names for SOAP nodes, outside the default (SOAP v1.2 1 2.2), SHALL NOT be defined for this service expression.

Additional implementation guidance is available in: Implementation Guidelines for the Open Archives Initiative Protocol for Metadata Harvesting. http://www.openarchives.org/OAI/2.0/guidelines.htm with specific:

• Guidelines for Repository Implementers. http://www.openarchives.org/OAI/2.0/guidelines-repository.htm
• Guidelines for Harvestor Implementers. http://www.openarchives.org/OAI/2.0/guidelines-harvester.htm
• Guidelines for Aggregators, Caches and ProxiesGuidelines for Aggregators, Caches and Proxies. http://www.openarchives.org/OAI/2.0/guidelines.htm

An implementation is discussed in: Congia, S. et al. 2004. Applying SOAP to OAI-PMH. In Heery, R. & Lyon, L. (eds), Proceedings 8th European Conference on Research and Advanced Technology for Digital Libraries (ECDL2004)(3232), pages 411-420, Bath, UK. http://pubs.cs.uct.ac.za/archive/00000273/

Another implementation is available from the International Virtual Observatory Alliance, a consortium of astronomers: http://www.ivoa.net/twiki/bin/view/IVOA/RegistryInterface . The FRED WSDL has been based on that of IVOA.

Implementations MAY use the SOAP Extensibility Model (SOAP modules, SOAP features), as long as any such modules do not impinge on the functionality of the service expression.

Actual: None

Potential: The service expression could be used in a service usage model for federated metadata repositories. A federated metadata registry would be the client that would send harvests requests to the service implementations that provide harvest interfaces to the provider repositories in the federation. The requests would be used to gather the metadata used to populate the federation registry. The client would periodically harvest the repositories in the federation to obtain updates to the objects held in the repositories. The federated metadata registry could also provide a service implementation interface to allow other clients to harvest the metadata in the federation registry.

None

FRED Service Expression Name: harvest resource—oai-pmh basic, Vx.xx.
The harvest resource—oai-pmh basic service expression provides the same core functionality as the harvest resource—oai-pmh lom ws service expression. The harvest resource—oai-pmh basic service expression is not required to support LOM metadata, and communications are via HTTP request.

FRED Service Usage Model: registry federation, Vx.xx. Harvest (genre)
Is a part of the registry federation service usage model (genre based) and is used to gather metadata from the repositories and collections that participate in the federation to build the registry data used for discovery.

None

© Copyright, University of Southern Queensland 2007 and University of Memphis, 2007. This work is created as part of the Federated Repositories for Education (FRED) Project within the Australian ADL Partnership Laboratory. The FRED project is sponsored by the Australian Commonwealth Department of Education, Science and Training under the Framework for Open Learning Programme. The Australian ADL Partnership Laboratory is supported by the University of Southern Queensland

Attribute this work as:
Harvest Resource-oai-pmh lom ws Service Expression, authored and submitted by Nick Nicholas and Daniel Rehak on behalf of the Federated Repositories for Education (FRED) Project within the Australian ADL Partnership Laboratory, © University of Southern Queensland and the University of Memphis, 2007.

Last updated 14 October 2008