Localisation of iSAC in Germany and Italy: results are kept up!

The ePractice blog: discuss, praise, disagree.

ePractice.eu provides its members with a blog in which all registered users can post opinions, questions and links to news related to eGovernment, eInclusion and eHealth. Your point of view is what makes ePractice.eu relevant to other public administrators all over Europe, so feel free to post and...

25 May 2012 | 172 Visits | Rating: 3.6 (maximum:5)

Hi, we have the last extension of the study so that we know now how iSAC behaves with not only with longer questions, and we can have also the comparison with Citizens Attention Services (CAS) performance answering the same set of questions put forward to a full online Citizen Attention Service like iSAC and the simple search engines like Google, Google local (also called minigoogle), Yahoo and Bing that are used for searching keywords-based queries instead of trying to answer in full the questions of Citizens.

We have here added the cases of Prato (PRATO) in Italy, Bremerhaven (BIT) in Germany, and soon the European Institute of Woman Health (EIWH) in Ireland, to the existing cases of Terrassa (TERRASSA) in Catalonia, Saint Médard en Jalles (SMED and SMED2) in France.

PILOT	Google (local)	Google	iSAC	Yahoo	Bing	CAS	CAS 1 refinement	CAS 2 refinements
TERRASSA	65%	85%	100%	85%	85%	100%
SMED	35%	75%	100%		50%
SMED2	10%	62%	100%		24%	57%	86%	100%
PRATO	55%	60%	85%		20%	65%	95%	100%
BIT	20%	60%	90%		5%	95%

Again, iSAC beats search engines and get even stronger when questions are completely formulated (SMED2) not as a collection of keywords (SMED). In the case of Saint Médard en Jalles (SMED) rows we see the decay in the performance of Google (both minigoogle and the general Google) and Bing after well formulated questions (SME2 row).

About CAS, the phone service, is able to answer the 100% of questions only after 2 refinements, being only 57% of answering at a first shot in the case of CAS in Saint Medard. (SMED).

A similar pattern happens with PRATO and BIT, where iSAC beats the search engines with performance nearly to 100% but higher of 85%. In the case of PRATO, iSAC answers 85% that is much better than the first answer given by their walk-in CAS service. In the case of BIT the result of 90% is comparable to the 95% of their phone CAS service.

The outlier is EIWH as a result of 65% because of not localising the iSAC to the needs of the domain on questions about Health. In a future post on July we will show the EIWH results after the localisation, as well as the results of the last pilot on LINIA AMICA in ITALY.

These results enlightens us how iSAC can be used as the right front-office for frequently asked questions, matching results of CAS (the phone service), and being a valuable tool for reducing costs for the administrations and helping to relax administrative burdens to citizens.

The list of questions for BIT, PRATO and EIWH follow. For seeing the list of questions of SMED and TERRASSA please see previous posts.

PRATO

Table: PRATO iSAC versus other search facilities · long question

Question	Google (local)	google.it	iSAC	bing
dove posso registrare il mio cane?	0	0	1	0
avere incentivi per trasformare l'auto da benzina a gpl	0	1	1	0
avere agevolazioni per il risparmio energetico	1	0	0	0
dove vado per chiedere l'esenzione del ticket sanitario?	0	0	1	0
come faccio per avere il certificato di nascita?	1	1	1	0
vorrei fare domanda per la disoccupazione	0	0	1	0
dove devo andare per rinnovare la patente auto?	0	1	1	1
come faccio ad avere la tessera sanitaria?	1	1	1	0
come faccio a fare ricorso per le case popolari?	1	1	1	0
come faccio a fare attività fisica se sono anziano?	1	0	0	0
come fare per cambiare residenza?	1	1	1	1
a chi mi posso rivolgere per gli assegni familiari?	1	0	1	0
come fare per ottenere le agevolazioni per portatori di handicap?	0	1	0	0
ho bisogno di un passaporto	0	0	1	0
come fare per ricorrere ad una sanzione del codice della strada?	1	1	1	0
come iscrivo mio figlio all'asilo nido?	0	1	1	0
vorrei chiedere contributi per il pagamento dell'affitto	1	0	1	0
come fare la carta d'identità	1	1	1	1
come ottenere aiuti per le famiglie?	1	1	1	0
come devo fare per ottenere la cittadinanza italiana?	0	1	1	1
Result	55%	60%	85%	20%

Note: or the query in google.it and bing the word “Prato” was added

Table: PRATO iSAC versus SAC by telephone

Question	URP answer at the first question	first refinement	URP answer at the first refinement?	second refinement	URP answer at the second refinement?
dove posso registrare il mio cane?	0	qual è il numero di telefono?	1
avere incentivi per trasformare l'auto da benzina a gpl	1
avere agevolazioni per il risparmio energetico	1
dove vado per chiedere l'esenzione del ticket sanitario?	1
come faccio per avere il certificato di nascita?	1
vorrei fare domanda per la disoccupazione	1
dove devo andare per rinnovare la patente auto?	1
come faccio ad avere la tessera sanitaria?	1
come faccio a fare ricorso per le case popolari?	0	dove trovo il modulo?	1
come faccio a fare attività fisica se sono anziano?	0	devo rivolgermi alla circoscrizione di residenza?	1
come fare per cambiare residenza?	1
a chi mi posso rivolgere per gli assegni familiari?	1
come fare per ottenere le agevolazioni per portatori di handicap?	0	ho bisogno del parcheggio handicap davanti casa	1	se avessi bisogno di altre informazioni sulle agevolazioni per i portatori di handicap?	1
ho bisogno di un passaporto	1
come fare per ricorrere ad una sanzione del codice della strada?	0	la sanzione è emessa dalla polizia stradale	1
come iscrivo mio figlio all'asilo nido?	1
vorrei chiedere contributi per il pagamento dell'affitto	1
come fare la carta d'identità	1
come ottenere aiuti per le famiglie?	1
come devo fare per ottenere la cittadinanza italiana?	0	sono cinese, sposato con italiana	1
Result	70%		30%		5%

In the case of PRATO these are the 20 most frequently asked questions. iSAC outperforms all other solutions. The percentage 85% can have a double reading:

It is positive in the sense that is higher than all other options
There is still space for improvement. Several concrete actions are possible (such as improve semantic network, improve positioning and widen the scope of search)

Comparing iSAC with the Citizen Attention via phone call, we can also detect a better performance of iSAC. Only one third of the questions could be answered by the personnel.

In the case of PRATO it should be taken into account that iSAC searches the web partially and another search facility co-exists with it.

BIT

Table: BIT iSAC versus other search facilities · long question

Question	Google (local)	google.de	iSAC	bing
wo kann ich meinen hund anmelden	0	1	1	0
wie ist der bearbeitungsstatus meines Persos?	0	1	1	0
wo kann ich ein gewerbe anmelden?	1	1	1	1
wo bekomme ich eine Baugenehmigung?	1	1	1	0
ich möchte meinen Wohnsitz anmelden	0	1	1	0
ich möchte mein arbeitslosengeld beantragen	0	0	1	0
ich brauche einen auszug aus der liegenschaftskarte	0	1	1	0
ich möchte eine abschrift beglaubigen lassen	0	0	1	0
wo kann ich elterngeld beantragen?	1	1	1	0
Wo bekomme ich meinen Führerschein	0	1	1	0
wo kann ich mich ins Grundbuch eintragen lassen	0	0	0	0
wo bekomme ich kindergeld?	0	0	1	0
ich möchte meine lohnsteuerkarte abholen	0	1	1	0
ich brauche einen reisepass	1	1	1	0
wo kann ich schüler-bafög beantragen	0	0	1	0
wo kann ich einen schwerbehindertenausweis beantragen	0	0	1	0
ich möchte wohngeld beantragen	0	1	1	0
wo bekomme ich wohnungsbauförderung	0	0	0	0
Wo bekomme ich einen Waffenschein	0	0	1	0
ich brauche eine becheinigung aus dem melderegister	0	1	1	0
Result	20%	60%	90%	5%

Table : BIT iSAC versus SAC by telephone

Question	CAS	Refinement
wo kann ich meinen hund anmelden	1	No additional questions
wie ist der bearbeitungsstatus meines Persos?	1	Asked for name
wo kann ich ein gewerbe anmelden?	1	Asked if first time registering a company
wo bekomme ich eine Baugenehmigung?	1	Asked what kind of building permit
ich möchte meinen Wohnsitz anmelden	1	No additional questions
ich möchte mein arbeitslosengeld beantragen	1	Asked for name
ich brauche einen auszug aus der liegenschaftskarte	1	No additional questions
ich möchte eine abschrift beglaubigen lassen	1	What kind of document should be accredited
wo kann ich elterngeld beantragen	1	No additional questions
Wo bekomme ich meinen Führerschein	1	No additional questions
wo kann ich mich ins Grundbuch eintragen lassen	1	No additional questions
wo bekomme ich kindergeld?	1	No additional questions
ich möchte meine lohnsteuerkarte abholen	1	Remark, that no income tax cards are issued any more
ich brauche einen reisepass	1	No additional questions
wo kann ich schüler-bafög beantragen	0	Redirect to other department, with no specific contact person
wo kann ich einen schwerbehindertenausweis beantragen	1	What kind of disability
ich möchte wohngeld beantragen	1	Reason for application
wo bekomme ich wohnungsbauförderung	1	No additional questions
Wo bekomme ich einen Waffenschein	1	No additional questions
ich brauche eine becheinigung aus dem melderegister	1	Reason, and data of person for which the certification is needed
Results	95%

In the case of BIT the results show an overall 90% of search success compared to other search facilities. Again:

It is positive in the sense that is higher than all other options
There is still space for improvement. Several concrete actions are possible (such as improve semantic network, improve positioning and widen the scope of search)

The comparative with the telephone service shows that the former has a very good response capacity.

In the case of BIT it should be taken into account that iSAC searches the web partially and another search facility co-exists with it.