Issue 1

1. Missing or Null Submissions

This notebook documents the analysis for Issue #1: Missing or Null Submissions in the Autonity Oracle data. It covers:

• What is this issue about?
  
• Why conduct this issue analysis?
  
• How to conduct this issue analysis?
  
• What are the results?

---

1.1 What Is This Issue About?

In the Oracle system, validators are expected to submit FX and token price data. However, some rows in the submission CSV files have missing or null values for one or more price fields. Examples include:

• Rows that only have a timestamp and validator address (no numeric values).
  
• Some currency pairs or token pairs are entirely missing in certain rows.
  
• Zero or placeholder values that suggest incomplete submissions.

This analysis investigates the frequency and patterns of these missing or null submissions.

---

1.2 Why Conduct This Issue Analysis?

• Reliability: Missing/null submissions can degrade the Oracle’s usefulness if data is incomplete when aggregated on-chain.
  
• Patterns & Evidence: Finding consistent patterns (which validators, which days/times) provides concrete evidence to the foundation and technical teams.
  
• Consistency: Understanding whether missing data spikes on weekends or for certain validators can be critical for consistency in data quality.

---

1.3 How to Conduct the Analysis?

Use Python with the Polars library (v1.24.0) to:

1. Load and preprocess Oracle submission CSV files.
2. Parse the timestamps from strings to actual datetimes (and derive day-of-week).
   
3. Define coverage modes:
   • ALL-FX mode: A row is considered valid if all FX and token columns are non-null.
     
   • ANY-FX mode: A row is considered valid if at least one FX or token column is non-null.
4. Compute coverage metrics:
   • For each timestamp, calculate how many validators are “present” in the CSV (i.e. have a row for that timestamp) and how many actually submit valid data (per the ALL-FX or ANY-FX definition).
   • Define a timestamp as “fully covered” if at least 90% of the present validators submitted valid data.
   • At the daily level, count how many timestamps met that 90% threshold (“full coverage”) versus those that did not (“incomplete coverage”).
5. Check weekend vs. weekday patterns by labeling days Monday=0 through Sunday=6, then aggregating coverage differences over weekends (Saturday=5, Sunday=6) vs. weekdays.

Below is the script to perform the analysis:

import polars as pl
import glob
import statistics
import warnings

warnings.filterwarnings("ignore")

def load_and_preprocess(submission_glob: str):
    """
    Loads Oracle Submission CSVs and returns a Polars DataFrame.
    """
    files = sorted(glob.glob(submission_glob))
    if not files:
        raise ValueError(f"No CSV files found matching pattern {submission_glob}")

    lf_list = []
    for f in files:
        lf_temp = pl.scan_csv(
            f,
            dtypes={"Timestamp": pl.Utf8},
            null_values=[""],
            ignore_errors=True,
        )
        lf_list.append(lf_temp)

    lf = pl.concat(lf_list)

    lf = lf.with_columns(
        pl.col("Timestamp")
        .str.strptime(pl.Datetime, strict=False)
        .alias("Timestamp_dt")
    )

    lf = lf.with_columns(
        [
            pl.col("Timestamp_dt").cast(pl.Date).alias("date_only"),
            pl.col("Timestamp_dt").dt.weekday().alias("weekday_num"),
        ]
    )

    return lf


def compute_coverage_metrics(
    df: pl.DataFrame,
    fx_cols: list[str],
    autonity_cols: list[str],
    use_all_fx_required: bool = True,
    coverage_threshold: float = 0.9,  # 90% coverage default
):
    """
    Compute coverage metrics, but treat a timestamp as "fully covered" only if
    at least `coverage_threshold` fraction of validators present at that
    timestamp provide valid data (ALL-FX or ANY-FX).
    """

    if use_all_fx_required:
        fx_expr = pl.fold(
            acc=pl.lit(True),
            function=lambda acc, x: acc & x,
            exprs=[pl.col(c).is_not_null() for c in fx_cols],
        ).alias("submitted_fx_data")

        autonity_expr = pl.fold(
            acc=pl.lit(True),
            function=lambda acc, x: acc & x,
            exprs=[pl.col(c).is_not_null() for c in autonity_cols],
        ).alias("submitted_autonity_data")
    else:
        fx_expr = pl.fold(
            acc=pl.lit(False),
            function=lambda acc, x: acc | x,
            exprs=[pl.col(c).is_not_null() for c in fx_cols],
        ).alias("submitted_fx_data")

        autonity_expr = pl.fold(
            acc=pl.lit(False),
            function=lambda acc, x: acc | x,
            exprs=[pl.col(c).is_not_null() for c in autonity_cols],
        ).alias("submitted_autonity_data")

    lf = df.lazy().with_columns([fx_expr, autonity_expr])

    if use_all_fx_required:
        condition_expr = pl.col("submitted_fx_data") & pl.col("submitted_autonity_data")
    else:
        condition_expr = pl.col("submitted_fx_data") | pl.col("submitted_autonity_data")

    lf = lf.with_columns(condition_expr.alias("any_submitted"))

    lf_per_addr = (
        lf.group_by(["Timestamp_dt", "Validator Address"])
        .agg(
            [
                pl.any("any_submitted").alias("any_submitted"),
                pl.any("submitted_fx_data").alias("fx_submitted"),
                pl.any("submitted_autonity_data").alias("autonity_submitted"),
            ]
        )
        .rename({"Timestamp_dt": "timestamp"})
    )

    lf_per_addr = lf_per_addr.with_columns(
        pl.col("timestamp").cast(pl.Date).alias("date_only")
    )

    lf_timestamp_coverage = (
        lf_per_addr.group_by(["date_only", "timestamp"])
        .agg(
            [
                pl.count("Validator Address").alias("validators_seen"),
                pl.sum("any_submitted").alias("num_submitted_any"),
                pl.sum("fx_submitted").alias("num_submitted_fx"),
                pl.sum("autonity_submitted").alias("num_submitted_autonity"),
            ]
        )
        .with_columns(
            (pl.col("num_submitted_any") / pl.col("validators_seen"))
            .fill_null(0.0)
            .alias("coverage_ratio")
        )
        .with_columns(
            (
                (pl.col("coverage_ratio") < coverage_threshold).cast(pl.Int8)
            ).alias("num_missing_any")
        )
    )

    df_timestamp_coverage = lf_timestamp_coverage.collect().sort(
        ["date_only", "timestamp"]
    )

    lf_day_coverage = (
        lf_timestamp_coverage.group_by("date_only")
        .agg(
            [
                pl.count("timestamp").alias("num_timestamps_that_day"),
                (pl.col("num_missing_any").eq(0).cast(pl.Int64))
                .sum()
                .alias("num_timestamps_full_coverage"),
                (pl.col("num_missing_any").gt(0).cast(pl.Int64))
                .sum()
                .alias("num_timestamps_incomplete_coverage"),
            ]
        )
        .with_columns(
            [
                (
                    pl.col("num_timestamps_full_coverage")
                    / pl.col("num_timestamps_that_day")
                ).alias("fraction_full_coverage"),
                (
                    pl.col("num_timestamps_incomplete_coverage")
                    / pl.col("num_timestamps_that_day")
                ).alias("fraction_incomplete_coverage"),
            ]
        )
    )

    df_day_coverage = lf_day_coverage.collect().sort("date_only")

    lf_missing_by_val = lf_per_addr.with_columns(
        (pl.col("any_submitted") == False).alias("is_missing")
    )

    lf_validator_missing_stats = (
        lf_missing_by_val.group_by("Validator Address")
        .agg(
            [
                pl.count("timestamp").alias("total_timestamps_encountered"),
                pl.sum("is_missing").alias("missing_count"),
            ]
        )
        .with_columns(
            (
                pl.col("missing_count") / pl.col("total_timestamps_encountered")
            ).alias("fraction_missing")
        )
    )

    df_validator_missing_stats = lf_validator_missing_stats.collect().sort(
        "fraction_missing", descending=True
    )

    mode_str = "ALL" if use_all_fx_required else "ANY"
    return df_timestamp_coverage, df_day_coverage, df_validator_missing_stats, mode_str


def check_weekend_patterns(df_timestamp_coverage: pl.DataFrame, df_source: pl.DataFrame):
    """
    Compare coverage on weekends (Sat=5, Sun=6) vs. weekdays (Mon-Fri=0..4).
    """
    df_day_and_week = (
        df_source.lazy()
        .group_by("date_only")
        .agg(
            [
                pl.first("weekday_num").alias("weekday_num"),
            ]
        )
        .collect()
    )

    df_cov_extended = df_timestamp_coverage.join(
        df_day_and_week, on="date_only", how="left"
    )

    df_weekday_cov = (
        df_cov_extended.lazy()
        .group_by("weekday_num")
        .agg(
            [
                pl.count("timestamp").alias("num_timestamps"),
                (pl.col("num_missing_any").eq(0).cast(pl.Int64))
                .sum()
                .alias("num_ts_full_coverage"),
            ]
        )
        .with_columns(
            [
                (
                    pl.col("num_ts_full_coverage") / pl.col("num_timestamps")
                ).alias("fraction_full_cov"),
            ]
        )
        .collect()
        .sort("weekday_num")
    )

    def weighted_fraction(df: pl.DataFrame) -> float:
        if df.is_empty():
            return 0.0
        total_ts = df["num_timestamps"].sum()
        if total_ts == 0:
            return 0.0
        full_cov = df["num_ts_full_coverage"].sum()
        return float(full_cov / total_ts)

    weekend_data = df_weekday_cov.filter(pl.col("weekday_num") >= 5)
    weekday_data = df_weekday_cov.filter(pl.col("weekday_num") < 5)

    weekend_cov = weighted_fraction(weekend_data)
    weekday_cov = weighted_fraction(weekday_data)

    return {
        "df_weekday_cov": df_weekday_cov,
        "weekend_fraction_full_cov": weekend_cov,
        "weekday_fraction_full_cov": weekday_cov,
    }


def analyze_missing_submissions_both_modes(submission_glob: str):
    """
    Main analysis function.
    """
    df_all_data = load_and_preprocess(submission_glob)

    fx_cols = [
        "AUD-USD Price","AUD-USD Confidence",
        "CAD-USD Price","CAD-USD Confidence",
        "EUR-USD Price","EUR-USD Confidence",
        "GBP-USD Price","GBP-USD Confidence",
        "JPY-USD Price","JPY-USD Confidence",
        "SEK-USD Price","SEK-USD Confidence",
    ]
    autonity_cols = [
        "ATN-USD Price","ATN-USD Confidence",
        "NTN-USD Price","NTN-USD Confidence",
        "NTN-ATN Price","NTN-ATN Confidence",
    ]

    (
        df_ts_cov_all,
        df_day_cov_all,
        df_val_missing_all,
        mode_str_all,
    ) = compute_coverage_metrics(df_all_data, fx_cols, autonity_cols, True)

    (
        df_ts_cov_any,
        df_day_cov_any,
        df_val_missing_any,
        mode_str_any,
    ) = compute_coverage_metrics(df_all_data, fx_cols, autonity_cols, False)

    weekend_info_all = check_weekend_patterns(df_ts_cov_all, df_all_data)
    weekend_info_any = check_weekend_patterns(df_ts_cov_any, df_all_data)

    return {
        "df_all_data": df_all_data,
        "ALL": {
            "df_timestamp_coverage": df_ts_cov_all,
            "df_day_coverage": df_day_cov_all,
            "df_validator_missing": df_val_missing_all,
            "weekend_info": weekend_info_all,
        },
        "ANY": {
            "df_timestamp_coverage": df_ts_cov_any,
            "df_day_coverage": df_day_cov_any,
            "df_validator_missing": df_val_missing_any,
            "weekend_info": weekend_info_any,
        },
    }

results = analyze_missing_submissions_both_modes(
    submission_glob="../submission-data/Oracle_Submission_*.csv"
)

---

1.4 What are the results?

Below are directly reference outcomes from the results dictionary obtained by executing the analysis. The results shown will automatically update when re-running this notebook with new or updated datasets.

Daily Coverage Analysis

ALL-FX Mode

# Display daily coverage for ALL-FX mode
results["ALL"]["df_day_coverage"]

all_day_cov = results["ALL"]["df_day_coverage"]["fraction_full_coverage"]
avg_all_cov = statistics.mean(all_day_cov) * 100
print(f"Average daily full coverage (ALL-FX mode): {avg_all_cov:.1f}%")
if avg_all_cov < 50:
    print("Coverage is relatively low, suggesting many validators frequently miss submitting complete data.")
else:
    print("Coverage is reasonably good, indicating validators frequently submit complete data.")

Average daily full coverage (ALL-FX mode): 97.5%
Coverage is reasonably good, indicating validators frequently submit complete data.

The table and statistics above summarize how frequently all validators submitted complete data each day.

ANY-FX Mode

# Display daily coverage for ANY-FX mode
results["ANY"]["df_day_coverage"]

any_day_cov = results["ANY"]["df_day_coverage"]["fraction_full_coverage"]
avg_any_cov = statistics.mean(any_day_cov) * 100
coverage_difference = avg_any_cov - avg_all_cov

print(f"Average daily full coverage (ANY-FX mode): {avg_any_cov:.1f}%")
print(f"Coverage difference between ANY-FX and ALL-FX modes: {coverage_difference:.1f}%")

if coverage_difference > 20:
    print("A substantial coverage improvement in ANY-FX mode indicates validators frequently provide partial submissions rather than complete ones.")
else:
    print("The small difference suggests that validators typically provide complete submissions or none at all.")

Average daily full coverage (ANY-FX mode): 97.5%
Coverage difference between ANY-FX and ALL-FX modes: 0.0%
The small difference suggests that validators typically provide complete submissions or none at all.

This comparison highlights the impact of submission requirements (complete vs. partial) on coverage.

Weekend vs. Weekday Coverage

weekend_cov = results["ALL"]["weekend_info"]["weekend_fraction_full_cov"] * 100
weekday_cov = results["ALL"]["weekend_info"]["weekday_fraction_full_cov"] * 100
print(f"Weekend coverage: {weekend_cov:.1f}%, Weekday coverage: {weekday_cov:.1f}%")

if weekend_cov > weekday_cov + 5:
    print("Significantly better coverage on weekends; potential scheduling issues on weekdays.")
elif weekday_cov > weekend_cov + 5:
    print("Significantly better coverage on weekdays; validators might be inactive or less reliable during weekends.")
else:
    print("No major difference in coverage between weekends and weekdays; submission patterns appear relatively uniform.")

Weekend coverage: 0.0%, Weekday coverage: 97.5%
Significantly better coverage on weekdays; validators might be inactive or less reliable during weekends.

List of all Validators and their Missing Rates

df_all_missing = results["ALL"]["df_validator_missing"]

for row in df_all_missing.iter_rows(named=True):
    addr = row["Validator Address"]
    total = row["total_timestamps_encountered"]
    missing = row["missing_count"]
    fraction = row["fraction_missing"] * 100
    print(
        f"Validator {addr}: total={total}, missing={missing}, fraction_missing={fraction:.1f}%"
    )

Validator 0x3fe573552E14a0FC11Da25E43Fef11e16a785068: total=2880, missing=2880, fraction_missing=100.0%
Validator 0xdF239e0D5b4E6e820B0cFEF6972A90893c2073AB: total=2880, missing=2880, fraction_missing=100.0%
Validator 0xd625d50B0d087861c286d726eC51Cf4Bd9c54357: total=2856, missing=2856, fraction_missing=100.0%
Validator 0x26E2724dBD14Fbd52be430B97043AA4c83F05852: total=2880, missing=2880, fraction_missing=100.0%
Validator 0x100E38f7BCEc53937BDd79ADE46F34362470577B: total=2876, missing=2876, fraction_missing=100.0%
Validator 0x4cD134001EEF0843B9c69Ba9569d11fDcF4bd495: total=2823, missing=52, fraction_missing=1.8%
Validator 0x19E356ebC20283fc74AF0BA4C179502A1F62fA7B: total=2833, missing=47, fraction_missing=1.7%
Validator 0xbfDcAF35f52F9ef423ac8F2621F9eef8be6dEd17: total=2833, missing=47, fraction_missing=1.7%
Validator 0xf34CD6c09a59d7D3d1a6C3dC231a7834E5615D6A: total=2834, missing=46, fraction_missing=1.6%
Validator 0x2928FE5b911BCAf837cAd93eB9626E86a189f1dd: total=2829, missing=44, fraction_missing=1.6%
Validator 0x01F788E4371a70D579C178Ea7F48E04e8B2CD743: total=2837, missing=43, fraction_missing=1.5%
Validator 0x6747c02DE7eb2099265e55715Ba2E03e8563D051: total=2840, missing=40, fraction_missing=1.4%
Validator 0x831B837C3DA1B6c2AB68a690206bDfF368877E19: total=2863, missing=17, fraction_missing=0.6%
Validator 0xd61a48b0e11B0Dc6b7Bd713B1012563c52591BAA: total=2873, missing=7, fraction_missing=0.2%
Validator 0x6a395dE946c0493157404E2b1947493c633f569E: total=2874, missing=6, fraction_missing=0.2%
Validator 0x5E17e837DcBa2728C94f95c38fA8a47CB9C8818F: total=2876, missing=4, fraction_missing=0.1%
Validator 0x984A46Ec685Bb41A7BBb2bc39f80C78410ff4057: total=2877, missing=3, fraction_missing=0.1%
Validator 0x1476A65D7B5739dE1805d5130441A94022Ee49fe: total=2462, missing=1, fraction_missing=0.0%
Validator 0x00a96aaED75015Bb44cED878D927dcb15ec1FF54: total=2866, missing=1, fraction_missing=0.0%
Validator 0x7232e75a8bFd8c9ab002BB3A00eAa885BC72A6dd: total=2877, missing=1, fraction_missing=0.0%
Validator 0xEf0Ba5e345C2C3937df5667A870Aae5105CAa3a5: total=2879, missing=1, fraction_missing=0.0%
Validator 0xBE287C82A786218E008FF97320b08244BE4A282c: total=2879, missing=1, fraction_missing=0.0%
Validator 0x8f91e0ADF8065C3fFF92297267E02DF32C2978FF: total=2879, missing=1, fraction_missing=0.0%
Validator 0xDF2D0052ea56A860443039619f6DAe4434bc0Ac4: total=2879, missing=1, fraction_missing=0.0%
Validator 0x23b4Be9536F93b8D550214912fD0e38417Ff7209: total=2880, missing=0, fraction_missing=0.0%
Validator 0x551f3300FCFE0e392178b3542c009948008B2a9F: total=2880, missing=0, fraction_missing=0.0%
Validator 0x59031767f20EA8F4a3d90d33aB0DAA2ca469Fd9a: total=2880, missing=0, fraction_missing=0.0%
Validator 0xD9fDab408dF7Ae751691BeC2efE3b713ba3f9C36: total=2880, missing=0, fraction_missing=0.0%
Validator 0x1Be7f70BCf8393a7e4A5BcC66F6f15d6e35cfBBC: total=2880, missing=0, fraction_missing=0.0%
Validator 0xBBf36374eb23968F25aecAEbb97BF3118f3c2fEC: total=2880, missing=0, fraction_missing=0.0%
Validator 0xE9FFF86CAdC3136b3D94948B8Fd23631EDaa2dE3: total=2880, missing=0, fraction_missing=0.0%
Validator 0xc5B9d978715F081E226cb28bADB7Ba4cde5f9775: total=2879, missing=0, fraction_missing=0.0%
Validator 0x3AaF7817618728ffEF81898E11A3171C33faAE41: total=2874, missing=0, fraction_missing=0.0%
Validator 0xcdEed21b471b0Dc54faF74480A0E700fCc42a7b6: total=2880, missing=0, fraction_missing=0.0%
Validator 0x94d28f08Ff81A80f4716C0a8EfC6CAC2Ec74d09E: total=2880, missing=0, fraction_missing=0.0%
Validator 0x99E2B4B27BDe92b42D04B6CF302cF564D2C13b74: total=2880, missing=0, fraction_missing=0.0%
Validator 0x24915749B793375a8C93090AF19928aFF1CAEcb6: total=2880, missing=0, fraction_missing=0.0%
Validator 0x383A3c437d3F12f60E5fC990119468D3561EfBfc: total=2880, missing=0, fraction_missing=0.0%
Validator 0x22A76e194A49c9e5508Cd4A3E1cD555D088ECB08: total=2880, missing=0, fraction_missing=0.0%
Validator 0x9d28e40E9Ec4789f9A0D17e421F76D8D0868EA44: total=2880, missing=0, fraction_missing=0.0%
Validator 0xf10f56Bf0A28E0737c7e6bB0aF92f3DDad34aE6a: total=2880, missing=0, fraction_missing=0.0%
Validator 0x64F83c2538A646A550Ad9bEEb63427a377359DEE: total=2880, missing=0, fraction_missing=0.0%
Validator 0x9C7dAABb5101623340C925CFD6fF74088ff5672e: total=2880, missing=0, fraction_missing=0.0%
Validator 0x791A7F840ac11841cCB0FaA968B2e3a0Db930fCe: total=2880, missing=0, fraction_missing=0.0%
Validator 0x527192F3D2408C84087607b7feE1d0f907821E17: total=2880, missing=0, fraction_missing=0.0%
Validator 0xcf716b3930d7cf6f2ADAD90A27c39fDc9D643BBd: total=2880, missing=0, fraction_missing=0.0%
Validator 0xfD97FB8835d25740A2Da27c69762D74F6A931858: total=2880, missing=0, fraction_missing=0.0%
Validator 0x94470A842Ea4f44e668EB9C2AB81367b6Ce01772: total=2880, missing=0, fraction_missing=0.0%
Validator 0xDCA5DFF3D42f2db3C18dBE823380A0A81db49A7E: total=2880, missing=0, fraction_missing=0.0%
Validator 0xB5d8be2AB4b6d7E6be7Ea28E91b370223a06289f: total=2880, missing=0, fraction_missing=0.0%
Validator 0x197B2c44b887c4aC01243BDE7E4bBa8bd95BC3a8: total=2880, missing=0, fraction_missing=0.0%
Validator 0x718361fc3637199F24a2437331677D6B89a40519: total=2880, missing=0, fraction_missing=0.0%
Validator 0x358488a4EdCA493FCD87610dcd50c62c8A3Dd658: total=2880, missing=0, fraction_missing=0.0%
Validator 0x36142A4f36974e2935192A1111C39330aA296D3C: total=2880, missing=0, fraction_missing=0.0%
Validator 0x3597d2D42f8Fbbc82E8b1046048773aD6DDB717E: total=2880, missing=0, fraction_missing=0.0%
Validator 0xF9B38D02959379d43C764064dE201324d5e12931: total=2880, missing=0, fraction_missing=0.0%
Validator 0x5603caFE3313D0cf56Fd4bE4A2f606dD6E43F8Eb: total=2880, missing=0, fraction_missing=0.0%
Validator 0xE4686A4C6E63A8ab51B458c52EB779AEcf0B74f7: total=2880, missing=0, fraction_missing=0.0%
Validator 0xC1F9acAF1824F6C906b35A0D2584D6E25077C7f5: total=2880, missing=0, fraction_missing=0.0%
Validator 0x8584A78A9b94f332A34BBf24D2AF83367Da31894: total=2879, missing=0, fraction_missing=0.0%

Please note, total is the number of timestamp slots at which this validator was expected to submit. missing indicates how many of those slots were unfilled (i.e. the validator did not provide the required data for that timestamp). fraction_missing is the percentage of timestamps that were missing out of the total encountered.