unesco_reader.core ================== .. py:module:: unesco_reader.core .. autoapi-nested-parse:: Core functions for the unesco_reader package This module contains the core functions for the unesco_reader package. These functions are used to interact with the UIS API to get data, metadata, and available indicators and geo units, themes, and data versions. The module handles indicator and entity conversions and normalizes data for easy processing to dataframes. The module handles errors and logs hints from the API responses Attributes ---------- .. autoapisummary:: unesco_reader.core.logger Functions --------- .. autoapisummary:: unesco_reader.core._log_hints unesco_reader.core._convert_codes unesco_reader.core._convert_indicator_codes_to_code unesco_reader.core._convert_geo_units_to_code unesco_reader.core._normalize_footnotes unesco_reader.core._add_indicator_labels unesco_reader.core._add_geo_unit_labels unesco_reader.core.get_data unesco_reader.core.get_metadata unesco_reader.core._indicators_df unesco_reader.core.available_indicators unesco_reader.core.available_geo_units unesco_reader.core.available_themes unesco_reader.core.default_version unesco_reader.core.available_versions Module Contents --------------- .. py:data:: logger .. py:function:: _log_hints(response: dict) -> None If there are any hints in the response, log them If a response from the api contains hints, it means there are some issues with the request or the data. This function logs the hints as warnings. There may be multiple hints, so they are logged one by one. :param response: The response from the API .. py:function:: _convert_codes(indicators: str | list[str], mapper: dict) -> str | list[str] Convert names to their respective codes This function is used to convert geo units or indicators from names to their respective codes. If the name is already a code, it is left as is. :param indicators: The indicator name or list of indicator names to convert to codes :param mapper: The dictionary mapping names to codes :returns: The code or list of codes .. py:function:: _convert_indicator_codes_to_code(indicators: str | list[str]) -> str | list[str] Convert indicators to their respective codes This function converts the indicator names to their respective codes. If the indicator is already a code, it is left as is. If the indicator is not found, it is left as is and will be handled as an error by the API. :param indicators: The indicator name or list of indicator names to convert to codes :returns: The indicator code or list of indicator codes .. py:function:: _convert_geo_units_to_code(geo_units: str | list[str]) -> str | list[str] Convert geo units to their respective codes This function converts the geo unit names to their respective codes. If the geo unit is already a code, it is left as is. If the geo unit is not found, it is left as is and will be handled as an error by the API. :param geo_units: The geo unit name or list of geo unit names to convert to codes :returns: The geo unit code or list of geo unit codes .. py:function:: _normalize_footnotes(data: list[dict]) -> list[dict] Normalize the footnotes column The footnotes can have 1 or multiple records with keys - "type", "subtype", "value". eg: {... 'footnotes': [{'type': 'Source', 'subtype': 'Data sources', 'value': "Country's submission to UIS Survey of Formal Education Questionnaire A"}], ... } This function normalizes the footnotes into a single string for a dataframe column with the structure: "type, subtype: value" eg: "Source, Data sources: Country's submission to UIS Survey of Formal Education Questionnaire A" For multiple footnotes, the normalized string is concatenated with a semicolon. .. py:function:: _add_indicator_labels(data: list[dict]) -> list[dict] Add indicator labels to the data :param data: The data to which to add the indicator labels :returns: The data with the indicator labels added .. py:function:: _add_geo_unit_labels(data: list[dict]) -> list[dict] Add geo unit labels to the data. For regions, add both the region name and the region group :param data: The data to which to add the geo unit labels :returns: The data with the geo unit labels added .. py:function:: get_data(indicator: str | list[str] | None = None, geoUnit: str | list[str] | None = None, start: int | None = None, end: int | None = None, labels: bool = False, geoUnitType: unesco_reader.config.GeoUnitType | None = None, footnotes: bool = False, *, raw: bool = False, version: str | None = None) -> pandas.DataFrame | list[dict] Get UIS data Query the UIS API for data based on the given parameters. At least one indicator or one geo_unit must be provided. If only indicators are provided, data for all geographies is returned, and vice versa. To see available indicators or geographies, use the `available_indicators` or `available_geo_units` functions respectively. If both a geo_unit and geo_unit_type are provided, the geo_unit_type is ignored. :param indicator: The indicator code or name to request data for. If None, data for all indicators is returned. By default, None. To see all available indicators, use the `available_indicators` function. :param geoUnit: The geo unit code or name to request data for. If None, data for all geo units is returned. By default, None. To see all available geo units, use the `available_geo_units` function. :param start: The start year to request data for. Includes the year itself. Default is None, which returns the earliest available year. :param end: The end year to request data for. Includes the year itself. Default is None, which returns the latest available year. :param labels: If True, adds indicator and geo unit labels to the data. Default is False. :param geoUnitType: The type of geography to request data for. Allowed values are NATIONAL and REGIONAL. If geoUnit is provided, this parameter is ignored. Default is both national and regional data :param footnotes: If True, includes footnotes in the response. Default is False. :param raw: If True, returns the data as a list of dictionaries in the original format from the API. Default is False. :param version: The data version to use. Default uses the latest default version. :returns: A pandas DataFrame with the data or a list of dictionaries if raw=True. .. py:function:: get_metadata(indicator: str | list[str] | None = None, disaggregations: bool = False, glossaryTerms: bool = False, *, version: str | None = None) -> list[dict] Get the metadata for indicators Get the metadata for the given indicators. If no indicator is provided, metadata for all indicators is returned. Optionally include disaggregations and glossary terms in the response. :param indicator: The indicator code or name to get metadata for. If None, metadata for all indicators is returned. Default is None which returns metadata for all indicators. To see all available indicators, use the `available_indicators` function. :param disaggregations: Include disaggregations in the response. Default is False. :param glossaryTerms: Include glossary terms in the response. Default is False. :param version: The data version to use. Default uses the latest default version. :returns: A list of dictionaries with the metadata for the indicators .. py:function:: _indicators_df(indicators: list[dict]) -> pandas.DataFrame Return available indicators as a DataFrame. This function flattens the data for easy DataFrame conversion then returns the DataFrame. :param indicators: The list of indicators to convert to a DataFrame :returns: A pandas DataFrame with the available indicators .. py:function:: available_indicators(theme: str | list[str] | None = None, minStart: int | None = None, geoUnitType: unesco_reader.config.GeoUnitType | Literal['ALL'] | None = None, *, raw: bool = False, version: str | None = None) -> pandas.DataFrame | list[dict] Get available indicators This functions returns the available indicators from the UIS API with some basic information, including theme, time range, last data update, and total records. The data is filtered based on the given parameters. :param theme: Filter indicators for specific themes. Can be a single theme or a list of themes. Default returns all themes. Use the `available_themes` function to see all available themes. :param minStart: The earliest start year for the indicator data. Includes the start year itself. Default is None, which returns all available data. :param geoUnitType: The type of geography for which data is available. Default is None which does not filter and gets any available type. Allowed values are "NATIONAL" (country-level data), "REGIONAL" (regional-level data), "ALL" (both national and regional data), or None for all types. :param raw: If True, returns the data as a list of dictionaries in the original format from the API. Default is False. :param version: The data version to use. Default uses the latest default version. :returns: A pandas DataFrame with the available indicators or a list of dictionaries if raw=True. .. py:function:: available_geo_units(geoUnitType: unesco_reader.config.GeoUnitType | None = None, *, raw: bool = False, version: str | None = None) -> pandas.DataFrame | list[dict] Get available geo units Get all available geo units for a given API data version (or the current default version if no explicit version is provided), along with some basic information like the region group and type of geography. :param geoUnitType: The type of geography to request data for. Allowed values are NATIONAL and REGIONAL. Default is None which returns all available types. :param raw: If True, returns the data as a list of dictionaries in the original format from the API. Default is False. :param version: The data version to use. Default uses the latest default version. :returns: A pandas DataFrame with the available geo units or a list of dictionaries if raw=True. .. py:function:: available_themes(*, raw: bool = False) -> pandas.DataFrame | dict Get the available themes and basic information including latest update and description :param raw: If True, returns the data as a dictionary in the original format from the API. Default is False. .. py:function:: default_version() -> str Get the default data version :returns: The default data version string .. py:function:: available_versions(*, raw: bool = False) -> pandas.DataFrame | list[dict] Get available data versions and basic information including publication date and description :param raw: If True, returns the data as a list of dictionaries in the original format from the API. Default is False. :returns: A pandas DataFrame with the available versions or a list of dictionaries if raw=True.