1. CWA Part 4 - Evaluation and Recommendations
Contents
1.1. Purpose and scope
This part will focus on evaluating the outcome of the CWA and to perform some tests to it by performing the test data registrations. Obviously this can only happen as soon as a the first implementations are available. The deliverable is therefore split into two phases: In the first phase, which is available for CWA version 2 the document will contain a general description about the test data that will be used for evaluation. In CWA version 3, the document will contain a detailed description and the outcome of the test. Based on those results, parts 4.3 and 4.4 will be created. Following this approach, the current version of this deliverable contains a description of the test data only as well a description of the methodology in 4.1.
1.2. Test Registrations
1.2.1. Introduction and Methodology
In order to ensure the usability of the CWA concepts in real world scenarios, a test registration will be created and used as a way to evaluate the results. This will help the project team to find bottlenecks and to give practical tipps and experiences for users that want to apply the results for sharing resources. The evaluation will be performed by creating two tests. The first one is testing the outcome of part 1, which is an approach to resource sharing and the corresponding Protocol for the Syndication of Service Descriptions. The second test is focusing on testing the Federated Terminological Resource Network (TRN) as described in part 2 of the CWA.
1.2.2. Test Registration 1
1.2.2.1. Test Data Description
Test first test scenario will focus on the approach to resource sharing and the corresponding Protocol for the Syndication of Service Descriptions as described in part 1 of the CWA. It will perform this test by following the use case described below:
1.2.2.1.1. Precondition
- It is assumed that an eGovernment resource has been created and that this resource should be shared among different partners or in the public. IT is also assumed that the responsible partners are aware of eGov-Share and that they have read and understood the CWA specification.
1.2.2.1.2. Testing steps
- The eGovernment resource needs to be able to cope with the ontology defined in part 1a.
- The eGovernment need to be published so that they can be accessed with the syndication protocol described in part 1b
- A subscription needs to be tested with an external ATOM compatible program
1.2.2.1.3. Evaluation criteria
The following criteria will be checked:
- Correctness of response (semantically)
- Correctness of data format (syntactically)
- Speed of response
- Possibility of configuration
1.2.2.2. Test Registration Outcome
This section is based on the outcome of the CWA implementation and it can therefore not be included before CWA version 3.
1.2.3. Test Registration 2
1.2.3.1. Test Data Description
Test second test scenario will focus on the evaluation of the Terminological Resource Network (TRN) as described in part 2 of the CWA. It will perform this test by following the use case described below:
1.2.3.1.1. Precondition
- It is assumed that a specific term has been typed in by a user. What is needed in order to provide a broad result set is a list of synonyms for this term. The TRN is therefore used to provide this list of synonyms. It is using the underlying structure of networked terms as described in part 2 of the CWA.
1.2.3.1.2. Testing steps
- The TRN is contacted and the synonym search is invoked with term “traffic”.
- In order to pass the test correctly the TRN needs to answer with a list of synonyms that have been compiled by analyzing all terms and their relationship
- A possible result set might be: traffic [EN]; verkehr [DE]; verkeer [NL].
1.2.3.1.3. Evaluation criteria
The following criteria will be checked:
- Correctness of response (semantically)
- Correctness of data format (syntactically)
- Speed of response
- Possibility of configuration (e.g. how can the system be influenced to return only elements in the same language, etc.)
1.2.3.2. Test Registration Outcome
The test has finished successfully with the following details:
1.2.3.2.1. Correctness of response (semantically)
Accepted - The response of the system has been { traffic, verkehr, verkeer } as expected. It should be noticed that the TRN has only been storing a very limited number of entries in the test. In real world scenarios, terms should be imported from existing terminology repositories.
1.2.3.2.2. Correctness of data format (syntactically)
Accepted – The output format has been returned correctly and was compliant to the W3C specification:
<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>
<head>
<variable name='subject'/>
<variable name='name'/>
<variable name='language'/>
<variable name='languagecode'/>
</head>
<results>
<result>
<binding name='language'>
<uri>http://www.cen.eu/egovshare/trn.owl#enus</uri>
</binding>
<binding name='languagecode'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>US</literal>
</binding>
<binding name='subject'>
<uri>http://www.cen.eu/egovshare/trn.owl#Term20</uri>
</binding>
<binding name='name'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>Traffic</literal>
</binding>
</result>
<result>
<binding name='language'>
<uri>http://www.cen.eu/egovshare/trn.owl#dede</uri>
</binding>
<binding name='languagecode'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>DE</literal>
</binding>
<binding name='subject'>
<uri>http://www.cen.eu/egovshare/trn.owl#Term21</uri>
</binding>
<binding name='name'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>Verkehr</literal>
</binding>
</result>
<result>
<binding name='language'>
<uri>http://www.cen.eu/egovshare/trn.owl#nlnl</uri>
</binding>
<binding name='languagecode'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>NL</literal>
</binding>
<binding name='subject'>
<uri>http://www.cen.eu/egovshare/trn.owl#Term21</uri>
</binding>
<binding name='name'>
<literal datatype='http://www.w3.org/2001/XMLSchema#string'>Verkeer</literal>
</binding>
</result>
</results>
</sparql>
1.2.3.2.3. Speed of response
Accepted – Speed of response has been 1.1 seconds.
1.2.3.2.4. Possibility of configuration
Accepted – The output format can be specified easily with a parameter. However, it needs to be noticed that the supported formats are currently limited to one specific output format, so the possibility of configuration is rather theoretical.
1.3. An approach for Ensuring Continuous Operation
1.3.1. Continuous Operation Planning
1.3.1.1. Introduction
The results of eGov-Share require a stable and long-lasting environment in order to provide advantages in real-world scenarios. The reasons for this are simple:
Locations and data change. For example, services might become available or unavailable or they might change the location and scope. Therefore the part 1 results need continuous updates.
Terms and term networks change over time. New terms appear and old terms become obsolete. For example, new regulations and rules in the European economy might create new terms or might group different terms. Therefore the terminological resource network needs to be updated continuously to reflect reality.
Technology changes rapidly. Especially in the domain of semantics, IT-formats and technology standards change every minute. eGov-Share uses a format independent way of providing data in order to provide applications and users with different ways of accessing the system and of using its data. However, it is important to add support for future standards because otherwise, eGov-Share will not be accepted by users.
Technical maintenance is needed in order to keep systems alive and in order to register new users and services. Therefore a technical maintenance is required to ensure a working environment.
1.3.1.2. Missing Elements
In order to cope with those requirements, it is necessary to ensure a continuous operation of eGov-Share. This will, however, require a more detailed specification than the one that has been described by this working group. For example, access rights and user management needs to be specified, which is pout of scope of this deliverable. It is therefore required to create an detailed specification that considers all possible elements that are required for moving from a demonstrator / prototype implementation to a bullet-proof implementation that could be used in day-to-day situations. The roadmap that is listed in the next section will sketch a path towards reaching this destination.
1.3.2. Roles and Responsibilities
1.3.2.1. Description of Roles and Responsibilities
In order to ensure a continuous operation, the following roles and responsibilities may be identified: 
1.3.2.2. Financial Issues
In order to ensure a continuous operation, a solid financial planning is required. This is mainly necessary to fund people that are capable of maintaining the data and the environment and in order to cover the costs for servers and day-to-day operation. In general, the following instruments can be applied:
- National or international funding: This would at least be necessary in order to realize the implementation and also to foster the usage of the results in the first months.
- Usage fee of users: It might be possible to charge users with a small monthly usage fee
- Usage fee of providers: In contrast to the usage fee, it might also be possible to charge a fee from eGovernment institutes that want to use eGov-Share results for making their services more widely available.
- Complementary incomes e.g. from advertising
- Voluntary work: Especially contributors of service entries or terms might add their input on a voluntary base helping the eGov-Share system to quickly react to new requirements.
In order to realize eGov-Share it needs to be analyzed in detail, which of those instruments are accepted by the market. For example, a usage fee will probably lead to a significant reduction of users.
1.4. Outcomes, Recommendations and Roadmap
1.4.1. Report on Workshop Findings and Outcomes
The aim of this workshop has been to define and build a number of tools that can be used to share and federate information about eGovernment resource themselves, with the overall objective to facilitate access to and hence (re-)use of eGovernment resources. Current obstacles to widespread access to and reuse of eGovernment resources include:
- Lack of a comprehensive, yet easy to use standardized metadata schema for their description;
- Lack of easily accessible administrative terminology for use in these descriptions;
- Lack of awareness of cultural diversity issues including users' language skills and other semiotic faculties.
This workshop has concentrated on addressing the obstacles above by specifying
- a standardized minimum description of eGovernment resources (leveraging existing federated terminological assets);
- mechanisms for the exchange and notification of new/updated information about eGovernment resources;
- common agreed approaches to facilitate the discovery of, query of, access to, and information about eGovernment resources.
Those specifications have been performed in part 1, 2 and 3 of the CWA with the goal of keeping practical needs in mind and with the focus on providing hand-on specifications that are build on widely applied but nevertheless modern formats and technologies. The CWA has specified interfaces and has provided practical examples in order to make the usage as easy as possible and in order to avoid problems in implementations of the specifications. The eGov-Share workshop has also implemented two demonstrator applications that can be used to validate the capabilities of the system although the demonstrators have been limited in scope and functionality in order to concentrate on the main principles. The specifications that have been described by the workshop such as the ATOM based protocol or the terminological resource description can be used without modification. The more partners are participating in implementing and using the workshop outcome, the more beneficial the results will be. This is especially true for the Terminological Resource Network which is based on the participation of many different participants.
1.4.2. Recommendations and Roadmap
1.4.2.1. Missing Elements
Although the workshop has created a detailed number of helpful specifications lot of things have still to be specified in order to put the eGov-Share vision into a valuable practice. The reason is that the eGov-Share workshop has been realized in a way that it has concentrated on describing a few major important results in detail instead of describing a lot of different things in a superficial way. The most important things that need specification and clarification are:
- User Management and profile issues
- Access Rights definition
- Handling temporarily unavailable resources
- Managing data exchange between multiple resources
- Performing an automatic or semi-automatic purification of conflicts
The roadmap in the next subsection will therefore start with the specification of the missing details in a holistic eGov-Share vision.
1.4.2.2. Roadmap towards future activities after workshop end
In order to realize the eGov-Share vision, the eGov-Share workshop team suggests facilitating the further development of a holistic solution that is wrapped around the specification of this CWA. More precisely it is recommended to create a project dedicated towards realizing the missing elements described above and towards implementing and evaluating a solid implementation of all concepts and specifications. In summary we firmly believe that the following steps would lead to a solid path towards a practical realization of the eGov-Share visions:
Although the demonstrator implementation of part 1 and part 2 are very helpful to demonstrate the overall idea, they are far from being used in a productive environment and are limited to the core specifications. We therefore suggest the creation of a two step implementation process with a 6 monthly beta test involving some real world eGovernment resources.