Overview

Breaking away from traditional approaches, Statistics Centre - Abu Dhabi (SCAD) is pioneering an innovative register-based census. This ground-breaking method relies exclusively on administrative data sources, tapping into education systems, housing, and employment registers, etc.

The transition from the traditional census, which has long served decision-making and strategic planning purposes, reflects the pressing need for accurate and timely statistics amidst the Emirate's rapid growth and transformation over the past decade. The register-based census promises substantial benefits, including cost savings, enhanced accuracy, reduced respondent burden, and expedited delivery of census results.

This shift entails fundamental changes in how population data is defined, collected, and utilised. Aligning concepts tailored for the traditional census with data from administrative sources presents challenges in terms of completeness and coverage. Moreover, determining the resident population from registers poses a significant hurdle, as discrepancies among registers can obscure the actual population size.

In response to these challenges, SCAD has devised an innovative solution: the iterative register-based census. This approach employs an index-based methodology, offering a fresh perspective on existing datasets and facilitating comparisons across multiple sources. By meticulously cross-referencing a diverse array of data sources, this method provides a more comprehensive and nuanced understanding of demographic trends.

The success of the Abu Dhabi Census 2023 reflects the strong partnership between SCAD and entities from all sectors. Their ongoing commitment is vital for updating statistical records. This enables us to provide regular, comprehensive statistical insights that empower decision-makers and enhance development plans within the Emirate of Abu Dhabi.

Objectives and Purposes

The primary objective of the register-based census is to obtain a specific set of variables derived from administrative records related to individuals, families, households, buildings and dwellings. It also serves to identify those usually residing in the Emirate of Abu Dhabi, both Emiratis and non-Emiratis, and to connect individuals to their places of residence, such as private or collective housing.

Additionally, the census aims to generate standardised data and information about the Emirate. It seeks to enhance the data quality within the Abu Dhabi Data Ecosystem and establish a comprehensive reference record for future censuses.

The continuously updated statistical registers, created for Abu Dhabi Census 2023, provide the opportunity to capture more frequent data on the population and its characteristics, with annual, quarterly, or even monthly updates becoming realistic. SCAD’s long-term vision is to be able to provide real-time census information to Abu Dhabi's decision-makers.

Regulatory Framework

On the 28th of April 2008, in accordance with Law No. (7) of 2008 and its amendments, SCAD was established to become the official source of statistical data in the Emirate of Abu Dhabi. In 2021, the Centre was reorganised in accordance with Law No. (5). Under this law, SCAD was mandated to develop and organise statistical work in the Emirate of Abu Dhabi and produce, analyse, and publish all official statistics related to the Emirate. It is entrusted with coordinating the process of integration and harmonisation of statistics produced by other local government departments and entities and coordinating its work with these entities regarding statistical concepts and definitions.

The Abu Dhabi Census 2023 relied on collecting and integrating data and administrative information available at government entities and other service providers in the Emirate of Abu Dhabi. Service-level agreements were entered between SCAD and each of the partner entities that provided source data for the census registers. Arrangements were made for secure file transfers and the protection of confidential data.

The completion of Abu Dhabi Census 2023 was made possible through the support and collaboration with partners. Their commitment to updating their administrative registers was instrumental. The registers utilised during Abu Dhabi Census 2023 have laid the foundation for accomplishing census on demand.

Methodology

The transition from a traditional survey-based census to a register-based census required the development of a new methodology. For a register-based census to be successful, it is crucial to have a well-prepared register system and a clearly defined methodology in place.

To conduct its innovative census on demand, we have used an index-based methodology. This offered a different view of known datasets and allowed for the comparison of multiple data sources. This approach relied on indexes created by extensively cross-referencing a broad spectrum of data sources.

The iterative register-based census provided a new way to count everyone in the Emirate by using information that was already collected for other reasons, such as rental contracts, school registrations, and employment records. The key benefits of this methodology include the maintenance of up-to-date data, consolidation of information from diverse sources for a comprehensive population overview, and enhanced accuracy. Additionally, it allows for a more flexible way to incorporate new data where necessary, without compromising on the privacy and security of individual information.

We have developed index-based methodologies called Residency Index and Place of Living Index, which used various registers and data sources to validate and refine register data.

Primarily, the methodology examines individuals' activities or Signs of Life across different databases to verify their residency status, relationships, and connections to specific locations.

Drawing insights from successful practices by a number of countries around the world, the methodology experienced further development and refinement to align with the specific requirements encountered within the local context. By tailoring the methodology to address the details and distinctions of the emirate’s administrative systems, data sources, and demographic dynamics, it became better equipped to effectively fulfill its intended purpose.

The implementation of Machine Learning algorithms and complex predictive tools unfolded potentials of census that have never been possible before.

Determining the residency status of individuals was achieved through a Residency Index. The Index is a critical metric for determining whether an individual is considered usual resident, relying on specific parameters and conditions. The index was calculated annually, based on the Signs of Life and a predefined threshold.

The parameters involved in the Residency Index calculation were derived from a combination of data sources, including the Population Register and the Signs of Life data. The Population Register contained information about persons whose identity documents were registered, while the Signs of Life data provided insights into the activities or indicators associated with an individual's presence in the Emirate.

To ensure accuracy and reliability, the Residency Index was designed to adhere to specific conditions and constraints. These conditions were defined based on the legal and regulatory framework governing residency in the UAE. This included rules regarding obtention or renewal of legal residency for individuals, and the use of specific data sources to determine the permanent nature of residency.

The Residency Index, calculated annually, ranged from 0 to 1, reflecting the probability of an individual being usual resident in Abu Dhabi. The calculation incorporated the individual's residency status from the previous year, and the Signs of Life gathered during that timeframe.

The availability of common identifiers enabled the monitoring of movements, meaning individuals leaving and returning to the UAE, or inter-emirate movers and commuters traveling between their home emirate and other emirates.

While the general principles of indexing were established and model parameters were empirically evaluated, the methodology remained dynamic, with the feasibility of adding new data and new signs every quarter, depending on the accessibility of new information (including big data).

The accuracy of index-based outputs was continuously assessed through additional surveys, connected to regular surveys, such as the Household Income and Expenditure Survey and the Labour Force survey. In these comparative assessments, any estimation errors were disclosed alongside the results.

The continuous addition of new information contributed to ongoing improvements in the accuracy of the index-based estimates.

The four key components and calculations for the designing and building of the Residency Index are as follows:

• Signs of Life(SOL): SOL could be described as any sort of event, activity, or occurrence of an individual that shows his/ her interaction with the respective data sources such as rental datasets, school enrolment, employment, and other data sources that provide evidence of a person's presence or activity in the emirate.
• Weights: calculating the weight of every designated source of Signs of Life to incorporate its significance in determining the residency status of individuals.
• Threshold(): To decide an individual's residency status, a threshold is applied to the Residency Index. If an individual's Residency Index is greater than or equal to ' ,' they are considered usual residents for a given year*. If it is less than '' they are considered non-usual residents.
• Stability rate: parameters and to incorporate information about the previous residency status of individuals where is applied for previous residency index of the individual and is applied for the current calculated sum signs of life and

The Residency Index created a register of the Usual Resident Population, using other internal registers continuously updated within SCAD. The Real Estate Register, the Labour Force Register, and the Collective Dwellings Register provided critical and updated information to achieve census on demand.

The Real Estate Register (RER)

The Real Estate Register (RER) is a critical resource for accurately determining population residence locations and identifying houses. It also establishes regions and administrative divisions, such as districts and communities. In the 2023 Abu Dhabi census, the geography was defined by the administrative boundaries set by the competent authorities.

Components of the Real Estate Register

The RER consists of two main categories:

• Buildings Register: Details about each building.
• Units Register: Information on individual units within buildings.

Information Provided by the Real Estate Register

The RER provides comprehensive data, including:

• Number of Buildings: Total count of buildings.
• Building Composition: Structural details of buildings.
• Unit Characteristics: Information on the use and type of individual units.

This data is crucial for understanding the demographic and economic landscape of the area.

The Labour Force Register was built from administrative sources of the entities in concern with employment registration in the federal government sector, the local government sector, the private sector, the semi-governmental sector, and the free zones. The Labour Force Register supported Abu Dhabi Census 2023, and continues to support future censuses, as a periodically updated tool that provides a close to reality picture of the Emirate’s labour force and its dynamism.

The Labour Force Register included all workers in the Emirate of Abu Dhabi and those who contributed to its economic activity, regardless of their place of residence. This included those who worked as usual residents of the Emirate, and those who worked in the Emirate and are residents of other emirates. It also included citizens seeking work and retirees.

Collective Dwellings Register provides data on public and collective housing‭, ‬such as hotels‭, ‬hostels‭, ‬dormitories for university students‭, ‬staff accommodation‭, ‬and other places of worship and their families‭ (‬who have no other housing‭). ‬It was created from the records of the authorities concerned with licensing workers‭, ‬managing work camps‭, ‬hotels‭, ‬and others‭.‬

The data quality standards were gradually improved over the course of the census. New variables were derived from existing ones to provide the required insights and deeper analysis, ensuring that datasets were as complete and accurate as possible. In cooperation with partners and data providers, remarkable results in improving data quality standards (data accuracy, coverage, and timing accuracy) were achieved throughout the census process.

After ensuring the quality and completeness of all data sets according to the metadata model, the data was moved into the statistical census record layout. This step involved structuring the data in a way that facilitated data access and analysis. During this process, additional improvements were worked on and implemented to meet the requirements. The results were presented to the specialised statistical team at SCAD for final review and approval and to ensure that the implementation was consistent with the goals and objectives of the register-based census.

Privacy

Statistical confidentiality is a fundamental principle of official statistics. Our disclosure control methods contain various measures that are applied to data to eliminate or reduce the risk of disclosing information about individual statistical units, such as individuals and households.

Advanced technologies were used to ensure the governance and security of information and related data. An intensive ethics program was employed for dealing with data, and secure procedures were adhered to throughout the preparation and implementation phases.

• Data was received from partners through direct API integrations and secure file transfer approach, as per best practices, to avoid any data leakage during the transfer.
• Periodic security tests have been conducted to comply with all security controls and mitigate any vulnerabilities.

Data Area Alignment with Privacy Pillars

SCAD Elevates Data Privacy & Confidentiality Standards

Statistics Centre - Abu Dhabi (SCAD) proudly announces its unwavering commitment to the highest echelons of data privacy and confidentiality. Underpinned by a robust Data Privacy and Confidentiality Policy, SCAD sets the benchmark for ethical data handling practices, aligning with global standards and regulatory mandates. The following processes were followed in the Census:

SCAD carefully collects and manages data, adhering to rigorous protocols and legal frameworks to safeguard confidentiality. All collected data is strictly utilized for statistical / Census purposes, with employees bound by confidentiality agreements to prevent unauthorized disclosure or utilization for purposes beyond statistical analysis.

SCAD employs advanced techniques to maintain the highest standards for the use and disclosure of data. Access to data is meticulously regulated based on employees’ roles, with strict approval procedures in place for access beyond designated scopes.

SCAD upholds a comprehensive data security framework, employing cutting-edge technologies and administrative safeguards to protect data integrity against unauthorized access or breaches. Regular security assessments are conducted to identify and address potential vulnerabilities, ensuring data security remains a top priority.

SCAD adopts a systematic approach to data storage and retention, segregating raw data and retaining personal data only for the duration necessary to fulfill its intended purpose or as mandated by law.

SCAD maintains rigorous compliance with its Data Privacy and Confidentiality Policy and all applicable data protection laws. Annual reviews ensure policies and procedures remain aligned with evolving legal and regulatory landscapes, underscoring SCAD’s unwavering commitment to privacy and data integrity.

Presentation of data

These dashboards track change in the dataset over different time periods and provide users with a geographical perspective of the data, focusing on population patterns and trends according to spatial distribution.

• Demographic and Social Characteristics
• Economic Characteristics
• Buildings and Housing Characteristics

Statistical disclosure control is applied to census output to ensure no individual information will be released. Rounding was applied as an effective tabular data disclosure control method. This method has been adopted also by Spain, Singapore, New Zealand and Canada as disclosure control approaches for their last census round. For the Abu Dhabi Census 2023, all statistics are rounded to a multiple of 5. It helps preserving individuals’ privacy while having no effect on the accuracy.

Quality Framework

Abu Dhabi Census 2023 was executed in a comprehensive six-phase process that was initiated with specifying needs and data collection design in 2022 and concluded with the dissemination of the census results in 2024.

The data collected were either taken directly from our data warehouse, or from different sharing methods agreed with the data providers. Data from multiple sources were integrated and coordinated through Application Programming Interfaces (API), specialised applications, and a secure file transfer protocol. The variables were defined and then standardised using national standards.

The creation of the databases required the application of record linkages and matching techniques to integrate administrative data from different sources.

Two types of variables were identified:

1. Key variables: the variables that were important in census for the analysis.
2. Linking variables: the variables used for linking different data sources.

A comprehensive quality measurement framework was established and implemented to evaluate the data quality at each stage of the census project, encompassing collection, processing, and dissemination. This framework is designed and implemented to ensure the census data's accuracy, reliability, and utility for facilitating evidence-based decision-making.

• Relevance: the extent to which statistical data meets the needs of current and potential users.
• Accuracy: the extent to which the data captures any discrepancies and deficiencies in the received data and the values of relevant variables.
• Timeliness: the extent to which partner entities update data and the degree of adherence to the agreed-upon schedule, and the frequency (e.g., monthly, quarterly) of data submission.
• Comparability and coherence: the extent to which published statistics are consistent and coherent.
• Accessibility and coverage: this dimension aims to improve data transfer methods and coverage by ensuring datasets provided to SCAD meet the agreed-upon specifications.

A set of rules was formulated to check the validity and accuracy of the input data and the coverage rate. Any identified errors were reported to the data providers for rectification. This procedure remained constant throughout the census phases, involving iteration, experimentation, and immediate rectification of potential technical issues.

Accuracy and coverage rate of the data were measured using special tools, and the results shared periodically with data providers in data quality reports that pointed out the shortfalls and recommended corrective actions to improve data quality. Technical meetings were conducted with data providers to provide feedback on data quality standards, as per Data Delivery Index.

As part of the census process, databases consisting of separate registers were created for each statistical unit, such as individuals, households, and dwellings. Accomplishing the census on demand was grounded on the development of key statistical registers and specialised statistical registers. The key registers, or the base registers, that hold the fundamental information required for the census, are Population Register, Real Estate Register, and Establishment Register. While key registers act as the spine for census, other specific information is retrieved from specialised registers such as Labour Force Register, Collective Dwellings Register, and Education Register. In each phase of Abu Dhabi Census 2023, these registers were updated and improved.

Future Applications of Census Data

The current census process provides a valuable opportunity to register and monitor ongoing changes across various sectors of society, such as education and employment. This will contribute to enhancing strategic planning and developing the economic and social sectors in the Emirate of Abu Dhabi.

Real-Time Data

The anticipation is high for conducting real-time censuses and obtaining live data from authorities. This kind of data, as envisioned by SCAD, is expected to enhance efficiency and effectiveness in decision-making, offering a more precise depiction of the current reality, trends, and patterns.

This vision requires creating an integrated system by combining various technologies that enable the collection, and processing of data. To ensure the speed of the data processing cycle, a robust database must be developed to store, analyse, and retrieve data efficiently.

Because real-time data systems are dynamic, providing future infrastructure is required to handle the evolving volume of data and user requests. Moreover, investing in research and development, and deploying innovation and advanced analytics tools to produce valuable data are required to provide clearer insights in real time to support decision-making. Keeping up to date with advancement in machine learning algorithms and predictive tools will lay the foundation to extract data about patterns and trends, enhancing a comprehensive understanding of the current situation and more accurately forecasting and foresighting the future.

Census GPT

Plans are in place to set up a Census GPT, using Large Language Models to improve data presentation and user experience.

Developing Interactive Statistics Dashboards and chat will provide users with detailed and analytical instant results that address their inquiries. These results are presented comprehensively for better analysis and effective decision-making. Furthermore, self-service tools and automatic insights will provide flexibility to users, enabling them to create personalized dashboards and get self-generated reports with recommendations within seconds.

History of Abu Dhabi Census

Since its foundation in 1971, the UAE has conducted five traditional censuses. These censuses, covering all seven Emirates of the UAE, were conducted in 1975, 1980, 1985, 1995 and 2005.

In addition to the census collections for the UAE, Abu Dhabi Emirate conducted three independent censuses in 2001, 2011 and 2023. The most recent one is the first census conducted entirely using administrative data.

In 2001 and 2011, censuses were traditional, except that the census in 2011 employed a face-to-face data collection method in which the interviewer used a tablet to record answers given. Some part of the population were collected administratively.

The register-based census is founded on the principle of generating census data and insights by harnessing a plethora of civil and administrative records. These records serve as repositories of information regarding statistical units, such as individuals, households, buildings and dwellings, capturing variables defined for administrative purposes.

Central to this approach is the linkage of data through unique identifying variables assigned for each individual, household, building and dwelling, facilitating the seamless integration of information across diverse records.

These administrative records originate from governmental and semi-governmental entities that provide services in Abu Dhabi. Applying the method of register-based census saves time and resources, providing opportunities for cost reduction, reducing the burden of the respondent, minimising low responses in the field, minimising human error, and enhancing privacy.

Moreover, these administrative registers constitute a reservoir of data, utilised to produce a statistical register that is characterised by accuracy and quality. This allows for up-to-date data for decision-makers.

FAQs

Statistical Data Revision Policy

Purpose

This policy establishes the governing boundaries for the revision of official statistical outputs disseminated by the Statistics Centre – Abu Dhabi (SCAD) and other Producers of Official Statistics (OPOS). It ensures that revisions are conducted transparently, consistently, and in accordance with internationally recognized statistical standards. The purpose is to enhance the credibility, reliability, and accuracy of statistical outputs and to maintain user confidence in the data products issued by SCAD.

Scope

This policy is applicable to all statistical data and outputs produced by SCAD & Other Producers of Official Statistics (OPOS) and disseminated by SCAD, regardless of format (electronic or print). It governs the revision of any data previously published, including but not limited to routine, major, and unscheduled revisions. The policy applies across the entire statistical production lifecycle.

Definition of Revision

A revision refers to any post-publication amendment made to previously disseminated statistical information. Revisions are a standard component of high-quality statistical production and are undertaken to reflect improved data accuracy, updated methodologies, corrected errors, or newly available information.

Types of Revisions

Revisions are categorized into three types based on their nature, scope, and trigger.

Note: In statistical work, errors in source data and computations occasionally occur. While every effort is made to minimize errors, SCAD has a long-standing policy to correct errors as soon as they are detected and to inform users by means of a correction of a published text.

Revision Governance & Calendar

Archiving

All revised statistical outputs, including their prior versions, shall be archived in accordance with SCAD’s Data Retention, Archiving & Disposal Policy (DAG-POL-12). The archiving process ensures traceability, transparency, and accountability of all revisions made post-dissemination.

Compliance

SCAD and OPOS involved in statistical production must comply with this policy.